Jul 08 06:52:37.896759 ip-10-0-135-211 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 08 06:52:37.896769 ip-10-0-135-211 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 08 06:52:37.896775 ip-10-0-135-211 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 08 06:52:37.897014 ip-10-0-135-211 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 08 06:52:48.117272 ip-10-0-135-211 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 08 06:52:48.117303 ip-10-0-135-211 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9c9c531b2cf94275b8b75f26ddb56c05 --
Jul 08 06:54:25.472307 ip-10-0-135-211 systemd[1]: Starting Kubernetes Kubelet...
Jul 08 06:54:25.875812 ip-10-0-135-211 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 06:54:25.875812 ip-10-0-135-211 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 08 06:54:25.875812 ip-10-0-135-211 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 06:54:25.875812 ip-10-0-135-211 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 08 06:54:25.875812 ip-10-0-135-211 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 06:54:25.876498 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.875876    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 08 06:54:25.883315 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883300    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 06:54:25.883315 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883314    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883317    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883321    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883324    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883327    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883330    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883333    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883336    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883338    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883341    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883344    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883347    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883349    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883352    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883355    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883357    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883360    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883362    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883365    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883373    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 06:54:25.883384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883376    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883378    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883380    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883383    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883385    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883388    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883391    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883394    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883396    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883399    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883401    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883404    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883406    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883409    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883411    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883414    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883416    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883418    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883420    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 06:54:25.883888 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883424    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883427    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883430    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883432    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883436    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883440    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883443    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883446    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883448    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883451    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883453    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883456    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883459    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883462    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883464    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883468    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883470    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883473    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883475    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883478    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 06:54:25.884381 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883480    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883482    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883485    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883487    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883490    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883492    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883495    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883497    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883500    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883502    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883504    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883508    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883511    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883514    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883517    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883519    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883522    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883524    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883526    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883529    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 06:54:25.884871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883531    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883534    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883536    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883539    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883541    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883544    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883546    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883550    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883554    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883557    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883559    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883562    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883565    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883567    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883570    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883572    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883574    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883577    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.883579    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884280    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 06:54:25.885394 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884286    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884289    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884292    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884295    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884298    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884300    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884304    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884308    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884311    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884314    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884316    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884319    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884321    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884323    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884326    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884328    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884331    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884333    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884336    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884340    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 06:54:25.885871 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884343    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884345    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884348    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884350    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884353    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884355    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884357    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884360    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884363    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884365    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884367    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884370    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884372    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884375    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884378    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884380    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884384    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884387    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884389    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884392    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 06:54:25.886384 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884394    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884397    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884399    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884402    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884404    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884406    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884409    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884411    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884413    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884416    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884418    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884421    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884424    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884427    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884429    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884432    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884435    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884439    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884442    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 06:54:25.886867 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884445    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884448    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884450    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884453    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884456    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884458    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884462    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884465    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884468    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884470    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884473    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884475    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884477    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884480    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884482    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884484    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884487    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884489    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884492    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884494    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 06:54:25.887344 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884497    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884499    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884501    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884505    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884509    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884512    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884521    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884524    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884526    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884529    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884531    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884534    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884536    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884539    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884541    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884544    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884546    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884548    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884551    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884980    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 06:54:25.887854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884985    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884988    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884991    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884994    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884997    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.884999    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885002    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885004    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885007    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885009    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885012    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885014    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885017    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885036    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885039    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885041    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885044    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885047    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885056    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885059    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 06:54:25.888359 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885061    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885064    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885066    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885069    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885071    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885074    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885076    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885080    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885083    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885086    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885088    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885091    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885094    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885096    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885099    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885101    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885103    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885106    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885108    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885111    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 06:54:25.888884 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885113    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885118    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885121    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885124    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885126    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885129    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885131    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885134    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885137    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885139    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885142    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885150    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885152    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885155    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885157    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885159    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885162    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885165    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885168    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 06:54:25.889374 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885171    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885174    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885179    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885181    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885184    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885186    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885189    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885191    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885194    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885196    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885198    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885201    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885203    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885206    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885208    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885210    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885214    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885216    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885218    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885221    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 06:54:25.889854 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885223    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885226    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885234    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885237    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885239    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885247    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885250    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885252    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885255    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885257    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885259    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885262    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885264    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885267    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885271    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885273    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885276    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885278    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.885281    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885342    2557 flags.go:64] FLAG: --address="0.0.0.0"
Jul 08 06:54:25.890398 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885353    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885372    2557 flags.go:64] FLAG: --anonymous-auth="true"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885376    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885382    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885385    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885389    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885393    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885396    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885398    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885402    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885405    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885408    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885411    2557 flags.go:64] FLAG: --cgroup-root=""
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885414    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885417    2557 flags.go:64] FLAG: --client-ca-file=""
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885420    2557 flags.go:64] FLAG: --cloud-provider="external"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885423    2557 flags.go:64] FLAG: --cluster-dns="[]"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885433    2557 flags.go:64] FLAG: --cluster-domain=""
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885443    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885446    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885449    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885452    2557 flags.go:64] FLAG: --container-log-max-files="5"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885455    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 08 06:54:25.890876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885458    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885461    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885464    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885469    2557 flags.go:64] FLAG: --contention-profiling="false"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885472    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885474    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885478    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885480    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885484    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885487    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885489    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885492    2557 flags.go:64] FLAG: --enable-load-reader="false"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885495    2557 flags.go:64] FLAG: --enable-server="true"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885498    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885504    2557 flags.go:64] FLAG: --event-burst="100"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885507    2557 flags.go:64] FLAG: --event-qps="50"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885510    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885512    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885515    2557 flags.go:64] FLAG: --eviction-hard=""
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885518    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885521    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885523    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885526    2557 flags.go:64] FLAG: --eviction-soft=""
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885529    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885532    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 08 06:54:25.891451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885535    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885537    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885540    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885548    2557 flags.go:64] FLAG: --fail-swap-on="true"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885551    2557 flags.go:64] FLAG: --feature-gates=""
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885554    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885557    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885560    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885563    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885566    2557 flags.go:64] FLAG: --healthz-port="10248"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885570    2557 flags.go:64] FLAG: --help="false"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885573    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-135-211.ec2.internal"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885576    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885578    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885581    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885584    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885587    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885590    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885593    2557 flags.go:64] FLAG: --image-service-endpoint=""
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885595    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885598    2557 flags.go:64] FLAG: --kube-api-burst="100"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885601    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885604    2557 flags.go:64] FLAG: --kube-api-qps="50"
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885607    2557 flags.go:64] FLAG: --kube-reserved=""
Jul 08 06:54:25.892110 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885609    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885612    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885615    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885617    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885620    2557 flags.go:64] FLAG: --lock-file=""
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885624    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885627    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885629    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885635    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885637    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885640    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885643    2557 flags.go:64] FLAG: --logging-format="text"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885651    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885654    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885657    2557 flags.go:64] FLAG: --manifest-url=""
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885660    2557 flags.go:64] FLAG: --manifest-url-header=""
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885663    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885666    2557 flags.go:64] FLAG: --max-open-files="1000000"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885671    2557 flags.go:64] FLAG: --max-pods="110"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885674    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885677    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885680    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885683    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885685    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885688    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 08 06:54:25.892830 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885691    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885697    2557 flags.go:64] FLAG: --node-status-max-images="50"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885700    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885702    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885705    2557 flags.go:64] FLAG: --pod-cidr=""
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885708    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885712    2557 flags.go:64] FLAG: --pod-manifest-path=""
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885715    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885718    2557 flags.go:64] FLAG: --pods-per-core="0"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885721    2557 flags.go:64] FLAG: --port="10250"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885724    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885726    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a196bb0805eb8340"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885729    2557 flags.go:64] FLAG: --qos-reserved=""
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885733    2557 flags.go:64] FLAG: --read-only-port="10255"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885736    2557 flags.go:64] FLAG: --register-node="true"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885739    2557 flags.go:64] FLAG: --register-with-taints=""
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885742    2557 flags.go:64] FLAG: --registry-burst="10"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885744    2557 flags.go:64] FLAG: --registry-qps="5"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885747    2557 flags.go:64] FLAG: --reserved-cpus=""
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885749    2557 flags.go:64] FLAG: --reserved-memory=""
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885758    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885761    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885764    2557 flags.go:64] FLAG: --rotate-certificates="false"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885767    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885769    2557 flags.go:64] FLAG: --runonce="false"
Jul 08 06:54:25.893836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885773    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885776    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885779    2557 flags.go:64] FLAG: --seccomp-default="false"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885782    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885784    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885787    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885790    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885793    2557 flags.go:64] FLAG: --storage-driver-password="root"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885795    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885798    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885801    2557 flags.go:64] FLAG: --storage-driver-user="root"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885803    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885806    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885809    2557 flags.go:64] FLAG: --system-cgroups=""
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885811    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885816    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885819    2557 flags.go:64] FLAG: --tls-cert-file=""
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885822    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885829    2557 flags.go:64] FLAG: --tls-min-version=""
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885832    2557 flags.go:64] FLAG: --tls-private-key-file=""
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885835    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885838    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885841    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885843    2557 flags.go:64] FLAG: --v="2"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885847    2557 flags.go:64] FLAG: --version="false"
Jul 08 06:54:25.894514 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885850    2557 flags.go:64] FLAG: --vmodule=""
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885853    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.885856    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886002    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886005    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886008    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886010    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886014    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886016    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886018    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886035    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886039    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886043    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886046    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886049    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886051    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886054    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886057    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886059    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 06:54:25.895130 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886062    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886064    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886067    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886070    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886074    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886076    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886080    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886082    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886086    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886088    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886091    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886093    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886096    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886098    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886101    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886103    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886105    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886114    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886116    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886119    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 06:54:25.895611 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886122    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886125    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886127    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886130    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886132    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886135    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886137    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886139    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886142    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886144    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886147    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886149    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886151    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886154    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886156    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886159    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886161    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886163    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886166    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886168    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 06:54:25.896122 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886172    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886175    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886177    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886180    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886182    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886185    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886187    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886189    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886192    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886194    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886202    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886205    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886208    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886210    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886213    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886215    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886217    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886220    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886222    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886224    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 06:54:25.896604 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886227    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886229    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886232    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886234    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886236    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886239    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886241    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886243    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886246    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886248    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886250    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886253    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886256    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886259    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886261    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886264    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886266    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886269    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886271    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886273    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 06:54:25.897110 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886276    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 06:54:25.897590 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886278    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 06:54:25.897590 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.886282    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 06:54:25.897590 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.886898    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 06:54:25.897590 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.894388    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 08 06:54:25.897590 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.894401    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 08 06:54:25.897590 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.894420    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 08 06:54:25.897590 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.894426    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 08 06:54:25.897590 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894486    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 06:54:25.897590 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894490    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 06:54:25.897590 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894493    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 06:54:25.897590 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894496    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 06:54:25.897590 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894499    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 06:54:25.897590 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894502    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 06:54:25.897590 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894505    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894507    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894510    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894513    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894515    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894518    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894521    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894523    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894526    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894529    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894531    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894534    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894537    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894539    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894542    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894544    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894547    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894549    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894552    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894555    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 06:54:25.897946 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894557    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894560    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894562    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894565    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894568    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894570    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894573    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894575    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894577    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894580    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894582    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894585    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894589    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894592    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894595    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894598    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894600    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894603    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894605    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 06:54:25.898470 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894608    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894610    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894613    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894615    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894617    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894620    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894622    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894625    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894627    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894629    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894632    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894634    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894637    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894639    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894641    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894643    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894646    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894648    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894650    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 06:54:25.898934 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894653    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894655    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894657    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894660    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894662    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894664    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894667    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894669    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894672    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894674    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894676    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894679    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894681    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894684    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894686    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894689    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894693    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894697    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894700    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894702    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 06:54:25.899420 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894704    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894707    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894710    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894713    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894716    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894719    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894722    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894724    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894727    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894729    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894731    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894734    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894736    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894738    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894741    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 06:54:25.899903 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.894746    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894854    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894859    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894862    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894865    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894867    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894870    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894873    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894877    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894880    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894882    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894885    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894887    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894890    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894892    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894895    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894897    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894900    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894902    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894905    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 06:54:25.900294 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894907    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894909    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894912    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894914    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894916    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894919    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894921    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894923    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894925    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894928    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894931    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894933    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894935    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894938    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894940    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894942    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894945    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894947    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894949    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894952    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 06:54:25.900764 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894954    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894957    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894959    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894962    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894964    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894968    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894972    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894975    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894978    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894981    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894984    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894987    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894989    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894991    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894994    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894996    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.894999    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895001    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 06:54:25.901249 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895003    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895006    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895008    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895010    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895013    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895015    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895017    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895034    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895038    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895042    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895046    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895048    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895050    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895053    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895055    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895058    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895060    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895062    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895065    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895067    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 06:54:25.901674 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895069    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895075    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895078    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895080    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895082    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895085    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895087    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895090    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895092    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895095    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895098    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895100    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895103    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895105    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895107    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895111    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895113    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895115    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895118    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895120    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 06:54:25.902290 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895122    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 06:54:25.902768 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:25.895125    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 06:54:25.902768 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.895129    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 06:54:25.902768 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.895227    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 08 06:54:25.902768 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.898122    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 08 06:54:25.902768 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.899389    2557 server.go:1019] "Starting client certificate rotation"
Jul 08 06:54:25.902768 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.899482    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 06:54:25.902768 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.899516    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 06:54:25.902768 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.901935    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 06:54:25.903061 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.903042    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 06:54:25.914672 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.914652    2557 log.go:25] "Validated CRI v1 runtime API"
Jul 08 06:54:25.918508 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.918495    2557 log.go:25] "Validated CRI v1 image API"
Jul 08 06:54:25.919389 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.919375    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 08 06:54:25.922489 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.922467    2557 fs.go:135] Filesystem UUIDs: map[594e991c-8707-415f-9b1d-9629030bb378:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f7cad5f8-3490-4883-8417-c8246bd9727b:/dev/nvme0n1p3]
Jul 08 06:54:25.922541 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.922489    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 08 06:54:25.928030 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.927909    2557 manager.go:216] Machine: {Timestamp:2026-07-08 06:54:25.926625091 +0000 UTC m=+0.338702825 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100303 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2140dd0470bf9ed9f0a38119e519dd SystemUUID:ec2140dd-0470-bf9e-d9f0-a38119e519dd BootID:9c9c531b-2cf9-4275-b8b7-5f26ddb56c05 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:db:c3:e3:7e:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:db:c3:e3:7e:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:20:84:af:f1:37 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 08 06:54:25.928030 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.928010    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 08 06:54:25.928168 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.928093    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 08 06:54:25.929755 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.929739    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 06:54:25.930636 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.930613    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 08 06:54:25.930806 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.930642    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-211.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 08 06:54:25.931394 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.931384    2557 topology_manager.go:138] "Creating topology manager with none policy"
Jul 08 06:54:25.931430 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.931397    2557 container_manager_linux.go:309] "Creating device plugin manager"
Jul 08 06:54:25.931430 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.931404    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 06:54:25.932013 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.932001    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 06:54:25.933574 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.933565    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 08 06:54:25.936844 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.936829    2557 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 06:54:25.936949 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.936941    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 08 06:54:25.936982 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.936968    2557 kubelet.go:486] "Attempting to sync node with API server"
Jul 08 06:54:25.936982 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.936979    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 08 06:54:25.937052 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.936992    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 08 06:54:25.937052 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.937000    2557 kubelet.go:398] "Adding apiserver pod source"
Jul 08 06:54:25.937052 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.937007    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 08 06:54:25.937370 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.937359    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 06:54:25.937417 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.937376    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 06:54:25.939388 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.939374    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 08 06:54:25.941088 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.941076    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 08 06:54:25.941139 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.941105    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 08 06:54:25.942122 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.942112    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 08 06:54:25.942170 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.942126    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 08 06:54:25.942170 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.942136    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 08 06:54:25.942170 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.942141    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 08 06:54:25.942170 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.942149    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 08 06:54:25.942170 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.942154    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 08 06:54:25.942170 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.942161    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 08 06:54:25.942170 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.942167    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 08 06:54:25.942339 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.942174    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 08 06:54:25.942339 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.942179    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 08 06:54:25.942339 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.942186    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 08 06:54:25.942339 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.942191    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 08 06:54:25.943074 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.943063    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 08 06:54:25.943074 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.943072    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 08 06:54:25.946068 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.946052    2557 server.go:1268] "Started kubelet"
Jul 08 06:54:25.946199 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.946173    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 08 06:54:25.946257 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.946247    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 08 06:54:25.946418 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.946391    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 08 06:54:25.947047 ip-10-0-135-211 systemd[1]: Started Kubernetes Kubelet.
Jul 08 06:54:25.947610 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.947239    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-211.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 08 06:54:25.947610 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:25.947282    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-211.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 08 06:54:25.947610 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:25.947368    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 08 06:54:25.947610 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.947447    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 08 06:54:25.948673 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.948661    2557 server.go:310] "Adding debug handlers to kubelet server"
Jul 08 06:54:25.951284 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.951264    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p2dvt"
Jul 08 06:54:25.952418 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:25.952392    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 08 06:54:25.953006 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.952985    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 08 06:54:25.953006 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.952996    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 08 06:54:25.953173 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:25.953144    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-211.ec2.internal\" not found"
Jul 08 06:54:25.953346 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:25.952136    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-211.ec2.internal.18c03e0a31def40d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-211.ec2.internal,UID:ip-10-0-135-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-211.ec2.internal,},FirstTimestamp:2026-07-08 06:54:25.946006541 +0000 UTC m=+0.358084275,LastTimestamp:2026-07-08 06:54:25.946006541 +0000 UTC m=+0.358084275,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-211.ec2.internal,}"
Jul 08 06:54:25.953635 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.953614    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 08 06:54:25.953635 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.953619    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 08 06:54:25.953813 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.953646    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 08 06:54:25.953813 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.953710    2557 reconstruct.go:97] "Volume reconstruction finished"
Jul 08 06:54:25.953813 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.953719    2557 reconciler.go:29] "Reconciler: start to sync state"
Jul 08 06:54:25.953948 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.953907    2557 factory.go:55] Registering systemd factory
Jul 08 06:54:25.953948 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.953925    2557 factory.go:223] Registration of the systemd container factory successfully
Jul 08 06:54:25.954142 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.954125    2557 factory.go:153] Registering CRI-O factory
Jul 08 06:54:25.954142 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.954142    2557 factory.go:223] Registration of the crio container factory successfully
Jul 08 06:54:25.954300 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.954182    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 08 06:54:25.954300 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.954203    2557 factory.go:103] Registering Raw factory
Jul 08 06:54:25.954300 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.954218    2557 manager.go:1196] Started watching for new ooms in manager
Jul 08 06:54:25.954740 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.954724    2557 manager.go:349] Starting recovery of all containers
Jul 08 06:54:25.956973 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:25.956947    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 08 06:54:25.957127 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:25.957015    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-211.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 08 06:54:25.957758 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.957741    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-p2dvt"
Jul 08 06:54:25.966975 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.966834    2557 manager.go:354] Recovery completed
Jul 08 06:54:25.971259 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.971242    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 06:54:25.973101 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.973087    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 06:54:25.973162 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.973114    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 06:54:25.973162 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.973124    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasSufficientPID"
Jul 08 06:54:25.973526 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.973515    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 08 06:54:25.973526 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.973524    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 08 06:54:25.973667 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.973537    2557 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 06:54:25.973667 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.973546    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 06:54:25.974230 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.974217    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 06:54:25.974286 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.974241    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 06:54:25.974286 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.974250    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasSufficientPID"
Jul 08 06:54:25.974727 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.974717    2557 policy_none.go:49] "None policy: Start"
Jul 08 06:54:25.974760 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.974732    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 08 06:54:25.974760 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.974742    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 08 06:54:25.975707 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:25.975698    2557 policy_none.go:47] "Start"
Jul 08 06:54:26.017553 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.017538    2557 manager.go:337] "Starting Device Plugin manager"
Jul 08 06:54:26.018646 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.017571    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 08 06:54:26.018646 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.017580    2557 server.go:87] "Starting device plugin registration server"
Jul 08 06:54:26.018646 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.017755    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 08 06:54:26.018646 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.017764    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 08 06:54:26.018646 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.017817    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 08 06:54:26.018646 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.017880    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 08 06:54:26.018646 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.017888    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 08 06:54:26.018646 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.018414    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 08 06:54:26.018646 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.018451    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-211.ec2.internal\" not found"
Jul 08 06:54:26.102361 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.102328    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 08 06:54:26.103781 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.103761    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 08 06:54:26.103879 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.103799    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 08 06:54:26.103879 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.103822    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 08 06:54:26.103879 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.103867    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 08 06:54:26.106835 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.106812    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 06:54:26.118069 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.118057    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 06:54:26.118901 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.118887    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 06:54:26.118966 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.118918    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 06:54:26.118966 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.118933    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasSufficientPID"
Jul 08 06:54:26.118966 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.118952    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.129861 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.129817    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.129861 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.129835    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-211.ec2.internal\": node \"ip-10-0-135-211.ec2.internal\" not found"
Jul 08 06:54:26.143388 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.143371    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-211.ec2.internal\" not found"
Jul 08 06:54:26.204833 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.204814    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-211.ec2.internal"]
Jul 08 06:54:26.204888 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.204881    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 06:54:26.205561 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.205549    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 06:54:26.205631 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.205572    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 06:54:26.205631 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.205581    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasSufficientPID"
Jul 08 06:54:26.206689 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.206676    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 06:54:26.206835 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.206822    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.206885 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.206848    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 06:54:26.207339 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.207302    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 06:54:26.207339 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.207309    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 06:54:26.207339 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.207331    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 06:54:26.207466 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.207345    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasSufficientPID"
Jul 08 06:54:26.207466 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.207333    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 06:54:26.207466 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.207428    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasSufficientPID"
Jul 08 06:54:26.208364 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.208352    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.208420 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.208375    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 06:54:26.208942 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.208929    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 06:54:26.209008 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.208954    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 06:54:26.209008 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.208964    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeHasSufficientPID"
Jul 08 06:54:26.233667 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.233643    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-211.ec2.internal\" not found" node="ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.237940 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.237924    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-211.ec2.internal\" not found" node="ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.244273 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.244258    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-211.ec2.internal\" not found"
Jul 08 06:54:26.255544 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.255523    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39302b21a7247f89d983575aca27ff92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal\" (UID: \"39302b21a7247f89d983575aca27ff92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.255615 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.255547    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39302b21a7247f89d983575aca27ff92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal\" (UID: \"39302b21a7247f89d983575aca27ff92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.255615 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.255564    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bea9ecd901e1fb22501c8126e0da3e1b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-211.ec2.internal\" (UID: \"bea9ecd901e1fb22501c8126e0da3e1b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.344664 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.344642    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-211.ec2.internal\" not found"
Jul 08 06:54:26.356001 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.355981    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39302b21a7247f89d983575aca27ff92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal\" (UID: \"39302b21a7247f89d983575aca27ff92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.356078 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.356006    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39302b21a7247f89d983575aca27ff92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal\" (UID: \"39302b21a7247f89d983575aca27ff92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.356078 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.356051    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bea9ecd901e1fb22501c8126e0da3e1b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-211.ec2.internal\" (UID: \"bea9ecd901e1fb22501c8126e0da3e1b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.356171 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.356090    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bea9ecd901e1fb22501c8126e0da3e1b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-211.ec2.internal\" (UID: \"bea9ecd901e1fb22501c8126e0da3e1b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.356171 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.356090    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/39302b21a7247f89d983575aca27ff92-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal\" (UID: \"39302b21a7247f89d983575aca27ff92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.356171 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.356090    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/39302b21a7247f89d983575aca27ff92-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal\" (UID: \"39302b21a7247f89d983575aca27ff92\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.445417 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.445368    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-211.ec2.internal\" not found"
Jul 08 06:54:26.536932 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.536906    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.540868 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.540729    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:26.545478 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.545462    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-211.ec2.internal\" not found"
Jul 08 06:54:26.645961 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.645939    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-211.ec2.internal\" not found"
Jul 08 06:54:26.746434 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.746386    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-211.ec2.internal\" not found"
Jul 08 06:54:26.847051 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.847014    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-211.ec2.internal\" not found"
Jul 08 06:54:26.898407 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.898383    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 08 06:54:26.898891 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.898506    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 06:54:26.947846 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:26.947828    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-211.ec2.internal\" not found"
Jul 08 06:54:26.961897 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.961844    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 06:49:27 +0000 UTC" deadline="2028-04-07 09:21:42.569209 +0000 UTC"
Jul 08 06:54:26.961987 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:26.961898    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15338h27m15.607315774s"
Jul 08 06:54:27.048600 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:27.048574    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-211.ec2.internal\" not found"
Jul 08 06:54:27.065284 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:27.065250    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39302b21a7247f89d983575aca27ff92.slice/crio-bbb97a1478637ed00ca894aeb2b0c0c4b15380ffc786049b892b277f99483246 WatchSource:0}: Error finding container bbb97a1478637ed00ca894aeb2b0c0c4b15380ffc786049b892b277f99483246: Status 404 returned error can't find the container with id bbb97a1478637ed00ca894aeb2b0c0c4b15380ffc786049b892b277f99483246
Jul 08 06:54:27.066216 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:27.066195    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea9ecd901e1fb22501c8126e0da3e1b.slice/crio-b76622eb201ee264a96096cc93733b55fbe5a201288140f0ec37c5bec4d3a515 WatchSource:0}: Error finding container b76622eb201ee264a96096cc93733b55fbe5a201288140f0ec37c5bec4d3a515: Status 404 returned error can't find the container with id b76622eb201ee264a96096cc93733b55fbe5a201288140f0ec37c5bec4d3a515
Jul 08 06:54:27.070504 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.070488    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 06:54:27.093637 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.093615    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 06:54:27.106727 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.106684    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal" event={"ID":"39302b21a7247f89d983575aca27ff92","Type":"ContainerStarted","Data":"bbb97a1478637ed00ca894aeb2b0c0c4b15380ffc786049b892b277f99483246"}
Jul 08 06:54:27.107715 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.107696    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-211.ec2.internal" event={"ID":"bea9ecd901e1fb22501c8126e0da3e1b","Type":"ContainerStarted","Data":"b76622eb201ee264a96096cc93733b55fbe5a201288140f0ec37c5bec4d3a515"}
Jul 08 06:54:27.149864 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.149846    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 06:54:27.153259 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.153245    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:27.165117 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.165102    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 06:54:27.166135 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.166124    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-211.ec2.internal"
Jul 08 06:54:27.172429 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.172416    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 06:54:27.252963 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.252943    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 06:54:27.903599 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.903574    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 06:54:27.938702 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.938672    2557 apiserver.go:52] "Watching apiserver"
Jul 08 06:54:27.946115 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.946083    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 08 06:54:27.947450 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.947423    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-8wkwl","kube-system/global-pull-secret-syncer-7zf27","kube-system/konnectivity-agent-m5n5f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw","openshift-dns/node-resolver-6gzn7","openshift-image-registry/node-ca-d5qzh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal","openshift-multus/multus-additional-cni-plugins-crwf7","openshift-multus/network-metrics-daemon-r4hnb","kube-system/kube-apiserver-proxy-ip-10-0-135-211.ec2.internal","openshift-cluster-node-tuning-operator/tuned-stt62","openshift-multus/multus-9lcnh","openshift-network-diagnostics/network-check-target-h4bmr","openshift-network-operator/iptables-alerter-xj4hz"]
Jul 08 06:54:27.950144 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.950050    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:27.950222 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:27.950136    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:27.952196 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.952175    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:27.952269 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:27.952248    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:27.954078 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.954052    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 08 06:54:27.954290 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.954273    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m5n5f"
Jul 08 06:54:27.956916 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.956427    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:27.956916 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.956548    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 08 06:54:27.956916 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.956640    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 08 06:54:27.956916 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.956550    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mxgmc\""
Jul 08 06:54:27.958692 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.958662    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6gzn7"
Jul 08 06:54:27.958828 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.958808    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 08 06:54:27.959037 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.958867    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zxjl2\""
Jul 08 06:54:27.959106 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.958932    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 08 06:54:27.959106 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.959065    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 08 06:54:27.960731 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.960714    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d5qzh"
Jul 08 06:54:27.961142 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.961124    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 08 06:54:27.961683 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.961533    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lllc2\""
Jul 08 06:54:27.962177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.961907    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 08 06:54:27.962835 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.962807    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 08 06:54:27.963097 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.963078    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 08 06:54:27.963194 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.963113    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 08 06:54:27.963281 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.963261    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9l4f\" (UniqueName: \"kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f\") pod \"network-check-target-h4bmr\" (UID: \"8de691b9-d68b-4ae5-a0aa-47587f146d7e\") " pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:27.963366 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.963296    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:27.963366 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.963319    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3b35714-946f-4024-8c67-ca8e4994f3a2-agent-certs\") pod \"konnectivity-agent-m5n5f\" (UID: \"e3b35714-946f-4024-8c67-ca8e4994f3a2\") " pod="kube-system/konnectivity-agent-m5n5f"
Jul 08 06:54:27.963366 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.963342    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b62rw\""
Jul 08 06:54:27.963484 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.963378    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:27.963484 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.963343    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39d02a2c-8802-46ce-a765-576cca29608e-kubelet-config\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:27.963484 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.963420    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39d02a2c-8802-46ce-a765-576cca29608e-dbus\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:27.963484 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.963435    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3b35714-946f-4024-8c67-ca8e4994f3a2-konnectivity-ca\") pod \"konnectivity-agent-m5n5f\" (UID: \"e3b35714-946f-4024-8c67-ca8e4994f3a2\") " pod="kube-system/konnectivity-agent-m5n5f"
Jul 08 06:54:27.965669 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.965400    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:27.966295 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.966215    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 08 06:54:27.966295 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.966271    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 08 06:54:27.966545 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.966457    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 08 06:54:27.966947 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.966930    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 08 06:54:27.967089 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.967075    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tfn8z\""
Jul 08 06:54:27.967186 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.967165    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 08 06:54:27.967301 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.967283    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 08 06:54:27.967588 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.967570    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 08 06:54:27.967703 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.967687    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:27.967785 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:27.967766    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:27.969240 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.969211    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 08 06:54:27.969329 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.969261    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 08 06:54:27.969393 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.969339    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 08 06:54:27.969661 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.969626    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-brpzq\""
Jul 08 06:54:27.969948 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.969930    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 08 06:54:27.970181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.970161    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 06:54:27.970982 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.970537    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xj4hz"
Jul 08 06:54:27.972634 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.972615    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 08 06:54:27.973361 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.973063    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 08 06:54:27.973361 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.973095    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kvgbq\""
Jul 08 06:54:27.973361 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.973155    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 08 06:54:27.973572 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.973553    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:27.975751 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.975730    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fddqr\""
Jul 08 06:54:27.975861 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.975768    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 08 06:54:27.975991 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.975936    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 08 06:54:27.976697 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.976683    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:27.978925 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.978907    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tsx7q\""
Jul 08 06:54:27.979126 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.978979    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 08 06:54:27.996134 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:27.996113    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dxpwp"
Jul 08 06:54:28.001966 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:28.001948    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dxpwp"
Jul 08 06:54:28.055162 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:28.055144    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 08 06:54:29.740210 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740184    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-modprobe-d\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.740312 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740223    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-systemd\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.740312 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740242    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-run-netns\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.740312 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740256    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-var-lib-kubelet\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.740312 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740277    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-registration-dir\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.740312 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740299    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-tuned\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.740560 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740341    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-multus-cni-dir\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.740560 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740371    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd567957-630a-4c89-8890-7abcf95e7298-host\") pod \"node-ca-d5qzh\" (UID: \"fd567957-630a-4c89-8890-7abcf95e7298\") " pod="openshift-image-registry/node-ca-d5qzh"
Jul 08 06:54:29.740560 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740398    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-run-ovn\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.740560 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740422    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-run-systemd\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.740560 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740449    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87h99\" (UniqueName: \"kubernetes.io/projected/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-kube-api-access-87h99\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.740560 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740474    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-sys\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.740560 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740508    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-etc-kubernetes\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.740560 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740540    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd567957-630a-4c89-8890-7abcf95e7298-serviceca\") pod \"node-ca-d5qzh\" (UID: \"fd567957-630a-4c89-8890-7abcf95e7298\") " pod="openshift-image-registry/node-ca-d5qzh"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740575    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2ns8\" (UniqueName: \"kubernetes.io/projected/56214b24-626b-4361-965b-b6b26940ac5d-kube-api-access-p2ns8\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740601    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b951e1f-2e78-4d1f-af51-eedaabc4a02a-iptables-alerter-script\") pod \"iptables-alerter-xj4hz\" (UID: \"5b951e1f-2e78-4d1f-af51-eedaabc4a02a\") " pod="openshift-network-operator/iptables-alerter-xj4hz"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740626    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740647    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56214b24-626b-4361-965b-b6b26940ac5d-system-cni-dir\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740686    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-lib-modules\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740724    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-hostroot\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740775    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-slash\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-run-openvswitch\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740811    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56214b24-626b-4361-965b-b6b26940ac5d-os-release\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740824    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b951e1f-2e78-4d1f-af51-eedaabc4a02a-host-slash\") pod \"iptables-alerter-xj4hz\" (UID: \"5b951e1f-2e78-4d1f-af51-eedaabc4a02a\") " pod="openshift-network-operator/iptables-alerter-xj4hz"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740837    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740857    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-os-release\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740879    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-node-log\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740899    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-cni-netd\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.740965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740952    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3f86a51-e751-4c0d-9958-611e61cdf7b0-env-overrides\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.740992    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56214b24-626b-4361-965b-b6b26940ac5d-cnibin\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741032    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-socket-dir\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741107    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d573e55-4edc-47fb-abeb-7c58ddb30139-tmp\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741129    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3f86a51-e751-4c0d-9958-611e61cdf7b0-ovnkube-script-lib\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741145    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56214b24-626b-4361-965b-b6b26940ac5d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741160    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-kubelet\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741178    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741192    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-sysctl-conf\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741205    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-host\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741235    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-var-lib-cni-bin\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741267    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-run-netns\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741291    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741312    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-cni-bin\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741334    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741356    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-kubernetes\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.741551 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741385    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-var-lib-kubelet\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741433    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67vx9\" (UniqueName: \"kubernetes.io/projected/0d573e55-4edc-47fb-abeb-7c58ddb30139-kube-api-access-67vx9\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741463    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-cni-binary-copy\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:29.741475    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741488    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-run-k8s-cni-cncf-io\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:29.741541    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret podName:39d02a2c-8802-46ce-a765-576cca29608e nodeName:}" failed. No retries permitted until 2026-07-08 06:54:30.241512341 +0000 UTC m=+2.977857052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret") pod "global-pull-secret-syncer-7zf27" (UID: "39d02a2c-8802-46ce-a765-576cca29608e") : object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741557    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jxzp\" (UniqueName: \"kubernetes.io/projected/fd567957-630a-4c89-8890-7abcf95e7298-kube-api-access-9jxzp\") pod \"node-ca-d5qzh\" (UID: \"fd567957-630a-4c89-8890-7abcf95e7298\") " pod="openshift-image-registry/node-ca-d5qzh"
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741596    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39d02a2c-8802-46ce-a765-576cca29608e-kubelet-config\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741619    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-device-dir\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741643    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-var-lib-cni-multus\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741666    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-multus-daemon-config\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741690    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-run-multus-certs\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741714    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-etc-openvswitch\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741719    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/39d02a2c-8802-46ce-a765-576cca29608e-kubelet-config\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741770    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ztl5\" (UniqueName: \"kubernetes.io/projected/c3f86a51-e751-4c0d-9958-611e61cdf7b0-kube-api-access-8ztl5\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741805    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3b35714-946f-4024-8c67-ca8e4994f3a2-agent-certs\") pod \"konnectivity-agent-m5n5f\" (UID: \"e3b35714-946f-4024-8c67-ca8e4994f3a2\") " pod="kube-system/konnectivity-agent-m5n5f"
Jul 08 06:54:29.742293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741827    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-sys-fs\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741847    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-system-cni-dir\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741861    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-multus-socket-dir-parent\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741880    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3b35714-946f-4024-8c67-ca8e4994f3a2-konnectivity-ca\") pod \"konnectivity-agent-m5n5f\" (UID: \"e3b35714-946f-4024-8c67-ca8e4994f3a2\") " pod="kube-system/konnectivity-agent-m5n5f"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741904    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhsvw\" (UniqueName: \"kubernetes.io/projected/77090b45-e309-490a-973e-28c723351e73-kube-api-access-mhsvw\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741941    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-systemd-units\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.741978    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3f86a51-e751-4c0d-9958-611e61cdf7b0-ovn-node-metrics-cert\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742007    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56214b24-626b-4361-965b-b6b26940ac5d-cni-binary-copy\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742029    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56214b24-626b-4361-965b-b6b26940ac5d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742048    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-sysconfig\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742067    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-sysctl-d\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742109    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742145    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56214b24-626b-4361-965b-b6b26940ac5d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742145    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742174    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ss7n\" (UniqueName: \"kubernetes.io/projected/5b951e1f-2e78-4d1f-af51-eedaabc4a02a-kube-api-access-4ss7n\") pod \"iptables-alerter-xj4hz\" (UID: \"5b951e1f-2e78-4d1f-af51-eedaabc4a02a\") " pod="openshift-network-operator/iptables-alerter-xj4hz"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742199    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39d02a2c-8802-46ce-a765-576cca29608e-dbus\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:29.742940 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742223    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-multus-conf-dir\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.743457 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742247    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-var-lib-openvswitch\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.743457 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742269    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3f86a51-e751-4c0d-9958-611e61cdf7b0-ovnkube-config\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.743457 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742287    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9l4f\" (UniqueName: \"kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f\") pod \"network-check-target-h4bmr\" (UID: \"8de691b9-d68b-4ae5-a0aa-47587f146d7e\") " pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:29.743457 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742308    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-run\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.743457 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742345    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-cnibin\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.743457 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742360    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4f29e73-17e3-42cd-8bb5-980dfe3729c2-hosts-file\") pod \"node-resolver-6gzn7\" (UID: \"a4f29e73-17e3-42cd-8bb5-980dfe3729c2\") " pod="openshift-dns/node-resolver-6gzn7"
Jul 08 06:54:29.743457 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742374    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hcdc\" (UniqueName: \"kubernetes.io/projected/a4f29e73-17e3-42cd-8bb5-980dfe3729c2-kube-api-access-9hcdc\") pod \"node-resolver-6gzn7\" (UID: \"a4f29e73-17e3-42cd-8bb5-980dfe3729c2\") " pod="openshift-dns/node-resolver-6gzn7"
Jul 08 06:54:29.743457 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742381    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/39d02a2c-8802-46ce-a765-576cca29608e-dbus\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:29.743457 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742388    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-log-socket\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.743457 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742445    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndp7j\" (UniqueName: \"kubernetes.io/projected/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-kube-api-access-ndp7j\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.743457 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742464    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e3b35714-946f-4024-8c67-ca8e4994f3a2-konnectivity-ca\") pod \"konnectivity-agent-m5n5f\" (UID: \"e3b35714-946f-4024-8c67-ca8e4994f3a2\") " pod="kube-system/konnectivity-agent-m5n5f"
Jul 08 06:54:29.743457 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.742471    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4f29e73-17e3-42cd-8bb5-980dfe3729c2-tmp-dir\") pod \"node-resolver-6gzn7\" (UID: \"a4f29e73-17e3-42cd-8bb5-980dfe3729c2\") " pod="openshift-dns/node-resolver-6gzn7"
Jul 08 06:54:29.745409 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.745391    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e3b35714-946f-4024-8c67-ca8e4994f3a2-agent-certs\") pod \"konnectivity-agent-m5n5f\" (UID: \"e3b35714-946f-4024-8c67-ca8e4994f3a2\") " pod="kube-system/konnectivity-agent-m5n5f"
Jul 08 06:54:29.749444 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:29.749427    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 06:54:29.749537 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:29.749447    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 06:54:29.749537 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:29.749457    2557 projected.go:196] Error preparing data for projected volume kube-api-access-x9l4f for pod openshift-network-diagnostics/network-check-target-h4bmr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:29.749537 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:29.749500    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f podName:8de691b9-d68b-4ae5-a0aa-47587f146d7e nodeName:}" failed. No retries permitted until 2026-07-08 06:54:30.249487538 +0000 UTC m=+2.985832230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x9l4f" (UniqueName: "kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f") pod "network-check-target-h4bmr" (UID: "8de691b9-d68b-4ae5-a0aa-47587f146d7e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.842819    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-log-socket\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.842858    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndp7j\" (UniqueName: \"kubernetes.io/projected/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-kube-api-access-ndp7j\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.842883    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4f29e73-17e3-42cd-8bb5-980dfe3729c2-tmp-dir\") pod \"node-resolver-6gzn7\" (UID: \"a4f29e73-17e3-42cd-8bb5-980dfe3729c2\") " pod="openshift-dns/node-resolver-6gzn7"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.842895    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-log-socket\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.842907    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-modprobe-d\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.842929    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-systemd\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.842951    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-run-netns\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.842973    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-var-lib-kubelet\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.842995    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-registration-dir\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843014    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-tuned\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843036    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-multus-cni-dir\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843053    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-var-lib-kubelet\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843035    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-run-netns\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843077    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-systemd\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843057    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-modprobe-d\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843061    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd567957-630a-4c89-8890-7abcf95e7298-host\") pod \"node-ca-d5qzh\" (UID: \"fd567957-630a-4c89-8890-7abcf95e7298\") " pod="openshift-image-registry/node-ca-d5qzh"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843097    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd567957-630a-4c89-8890-7abcf95e7298-host\") pod \"node-ca-d5qzh\" (UID: \"fd567957-630a-4c89-8890-7abcf95e7298\") " pod="openshift-image-registry/node-ca-d5qzh"
Jul 08 06:54:29.843177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843097    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-registration-dir\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843132    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-run-ovn\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843159    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-run-systemd\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843134    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-multus-cni-dir\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843187    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87h99\" (UniqueName: \"kubernetes.io/projected/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-kube-api-access-87h99\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843201    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-run-systemd\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843206    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-run-ovn\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843211    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-sys\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843252    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-etc-kubernetes\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843253    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4f29e73-17e3-42cd-8bb5-980dfe3729c2-tmp-dir\") pod \"node-resolver-6gzn7\" (UID: \"a4f29e73-17e3-42cd-8bb5-980dfe3729c2\") " pod="openshift-dns/node-resolver-6gzn7"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843274    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd567957-630a-4c89-8890-7abcf95e7298-serviceca\") pod \"node-ca-d5qzh\" (UID: \"fd567957-630a-4c89-8890-7abcf95e7298\") " pod="openshift-image-registry/node-ca-d5qzh"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843288    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-etc-kubernetes\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843295    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2ns8\" (UniqueName: \"kubernetes.io/projected/56214b24-626b-4361-965b-b6b26940ac5d-kube-api-access-p2ns8\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843252    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-sys\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843340    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b951e1f-2e78-4d1f-af51-eedaabc4a02a-iptables-alerter-script\") pod \"iptables-alerter-xj4hz\" (UID: \"5b951e1f-2e78-4d1f-af51-eedaabc4a02a\") " pod="openshift-network-operator/iptables-alerter-xj4hz"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843365    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843390    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56214b24-626b-4361-965b-b6b26940ac5d-system-cni-dir\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843412    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-lib-modules\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.844118 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843435    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-hostroot\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843458    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-slash\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:29.843469    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843482    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-run-openvswitch\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843506    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56214b24-626b-4361-965b-b6b26940ac5d-os-release\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:29.843521    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs podName:77090b45-e309-490a-973e-28c723351e73 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:30.343503503 +0000 UTC m=+3.079848209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs") pod "network-metrics-daemon-r4hnb" (UID: "77090b45-e309-490a-973e-28c723351e73") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843550    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b951e1f-2e78-4d1f-af51-eedaabc4a02a-host-slash\") pod \"iptables-alerter-xj4hz\" (UID: \"5b951e1f-2e78-4d1f-af51-eedaabc4a02a\") " pod="openshift-network-operator/iptables-alerter-xj4hz"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843569    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/56214b24-626b-4361-965b-b6b26940ac5d-os-release\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843575    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843604    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-os-release\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843626    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-node-log\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843645    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd567957-630a-4c89-8890-7abcf95e7298-serviceca\") pod \"node-ca-d5qzh\" (UID: \"fd567957-630a-4c89-8890-7abcf95e7298\") " pod="openshift-image-registry/node-ca-d5qzh"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843649    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-cni-netd\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843673    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3f86a51-e751-4c0d-9958-611e61cdf7b0-env-overrides\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843687    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56214b24-626b-4361-965b-b6b26940ac5d-system-cni-dir\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843697    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-lib-modules\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843698    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56214b24-626b-4361-965b-b6b26940ac5d-cnibin\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.844686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843720    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843726    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b951e1f-2e78-4d1f-af51-eedaabc4a02a-host-slash\") pod \"iptables-alerter-xj4hz\" (UID: \"5b951e1f-2e78-4d1f-af51-eedaabc4a02a\") " pod="openshift-network-operator/iptables-alerter-xj4hz"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843735    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56214b24-626b-4361-965b-b6b26940ac5d-cnibin\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843745    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-node-log\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843737    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-socket-dir\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843796    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d573e55-4edc-47fb-abeb-7c58ddb30139-tmp\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843806    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-slash\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843816    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-run-openvswitch\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843822    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3f86a51-e751-4c0d-9958-611e61cdf7b0-ovnkube-script-lib\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843835    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-os-release\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843849    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56214b24-626b-4361-965b-b6b26940ac5d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843854    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-cni-netd\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843874    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-kubelet\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843871    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b951e1f-2e78-4d1f-af51-eedaabc4a02a-iptables-alerter-script\") pod \"iptables-alerter-xj4hz\" (UID: \"5b951e1f-2e78-4d1f-af51-eedaabc4a02a\") " pod="openshift-network-operator/iptables-alerter-xj4hz"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843890    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-hostroot\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843887    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-socket-dir\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843933    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-kubelet\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843961    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.845309 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843984    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-sysctl-conf\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.843997    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56214b24-626b-4361-965b-b6b26940ac5d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844003    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-host\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844028    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-var-lib-cni-bin\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844045    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-host\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844052    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-run-netns\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844083    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-var-lib-cni-bin\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844107    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-run-netns\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844136    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844148    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-etc-selinux\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844165    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-cni-bin\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844176    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-sysctl-conf\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844195    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844203    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-kubernetes\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844225    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-var-lib-kubelet\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844227    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-cni-bin\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844241    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-kubernetes\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.846368 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844248    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67vx9\" (UniqueName: \"kubernetes.io/projected/0d573e55-4edc-47fb-abeb-7c58ddb30139-kube-api-access-67vx9\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844271    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c3f86a51-e751-4c0d-9958-611e61cdf7b0-env-overrides\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844276    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-cni-binary-copy\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844302    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-run-k8s-cni-cncf-io\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844325    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jxzp\" (UniqueName: \"kubernetes.io/projected/fd567957-630a-4c89-8890-7abcf95e7298-kube-api-access-9jxzp\") pod \"node-ca-d5qzh\" (UID: \"fd567957-630a-4c89-8890-7abcf95e7298\") " pod="openshift-image-registry/node-ca-d5qzh"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844350    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-device-dir\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844374    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-var-lib-cni-multus\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844385    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c3f86a51-e751-4c0d-9958-611e61cdf7b0-ovnkube-script-lib\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844396    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-multus-daemon-config\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844384    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-run-k8s-cni-cncf-io\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844276    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-var-lib-kubelet\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844430    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-device-dir\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844432    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-run-multus-certs\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844454    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-var-lib-cni-multus\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844465    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-host-run-multus-certs\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844469    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-etc-openvswitch\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844499    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ztl5\" (UniqueName: \"kubernetes.io/projected/c3f86a51-e751-4c0d-9958-611e61cdf7b0-kube-api-access-8ztl5\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.847181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844525    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-sys-fs\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844546    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-system-cni-dir\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844569    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-multus-socket-dir-parent\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844594    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhsvw\" (UniqueName: \"kubernetes.io/projected/77090b45-e309-490a-973e-28c723351e73-kube-api-access-mhsvw\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844617    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-systemd-units\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844641    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3f86a51-e751-4c0d-9958-611e61cdf7b0-ovn-node-metrics-cert\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844676    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56214b24-626b-4361-965b-b6b26940ac5d-cni-binary-copy\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844702    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56214b24-626b-4361-965b-b6b26940ac5d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844727    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-sysconfig\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844746    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-system-cni-dir\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844770    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-sysctl-d\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844797    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844812    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-cni-binary-copy\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844826    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56214b24-626b-4361-965b-b6b26940ac5d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844814    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-multus-socket-dir-parent\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844864    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ss7n\" (UniqueName: \"kubernetes.io/projected/5b951e1f-2e78-4d1f-af51-eedaabc4a02a-kube-api-access-4ss7n\") pod \"iptables-alerter-xj4hz\" (UID: \"5b951e1f-2e78-4d1f-af51-eedaabc4a02a\") " pod="openshift-network-operator/iptables-alerter-xj4hz"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844893    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-multus-conf-dir\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.847814 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844917    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-var-lib-openvswitch\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844940    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3f86a51-e751-4c0d-9958-611e61cdf7b0-ovnkube-config\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844954    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-multus-daemon-config\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844705    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-systemd-units\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844975    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-run\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844999    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-cnibin\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.845013    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-multus-conf-dir\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.845023    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4f29e73-17e3-42cd-8bb5-980dfe3729c2-hosts-file\") pod \"node-resolver-6gzn7\" (UID: \"a4f29e73-17e3-42cd-8bb5-980dfe3729c2\") " pod="openshift-dns/node-resolver-6gzn7"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.845017    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-sysconfig\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.845060    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-etc-openvswitch\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.845108    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4f29e73-17e3-42cd-8bb5-980dfe3729c2-hosts-file\") pod \"node-resolver-6gzn7\" (UID: \"a4f29e73-17e3-42cd-8bb5-980dfe3729c2\") " pod="openshift-dns/node-resolver-6gzn7"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.845145    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-var-lib-openvswitch\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.844640    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-sys-fs\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.845064    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hcdc\" (UniqueName: \"kubernetes.io/projected/a4f29e73-17e3-42cd-8bb5-980dfe3729c2-kube-api-access-9hcdc\") pod \"node-resolver-6gzn7\" (UID: \"a4f29e73-17e3-42cd-8bb5-980dfe3729c2\") " pod="openshift-dns/node-resolver-6gzn7"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.845222    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-run\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.845312    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-cnibin\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.845369    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-sysctl-d\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.845417    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3f86a51-e751-4c0d-9958-611e61cdf7b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.848459 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.845544    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c3f86a51-e751-4c0d-9958-611e61cdf7b0-ovnkube-config\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.849228 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.846207    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56214b24-626b-4361-965b-b6b26940ac5d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.849228 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.846209    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56214b24-626b-4361-965b-b6b26940ac5d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.849228 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.846273    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56214b24-626b-4361-965b-b6b26940ac5d-cni-binary-copy\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.849228 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.847487    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0d573e55-4edc-47fb-abeb-7c58ddb30139-etc-tuned\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.849228 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.847498    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c3f86a51-e751-4c0d-9958-611e61cdf7b0-ovn-node-metrics-cert\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.854082 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.854047    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2ns8\" (UniqueName: \"kubernetes.io/projected/56214b24-626b-4361-965b-b6b26940ac5d-kube-api-access-p2ns8\") pod \"multus-additional-cni-plugins-crwf7\" (UID: \"56214b24-626b-4361-965b-b6b26940ac5d\") " pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.854743 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.854717    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87h99\" (UniqueName: \"kubernetes.io/projected/aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6-kube-api-access-87h99\") pod \"aws-ebs-csi-driver-node-7phvw\" (UID: \"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.855335 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.855311    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jxzp\" (UniqueName: \"kubernetes.io/projected/fd567957-630a-4c89-8890-7abcf95e7298-kube-api-access-9jxzp\") pod \"node-ca-d5qzh\" (UID: \"fd567957-630a-4c89-8890-7abcf95e7298\") " pod="openshift-image-registry/node-ca-d5qzh"
Jul 08 06:54:29.856306 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.856285    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67vx9\" (UniqueName: \"kubernetes.io/projected/0d573e55-4edc-47fb-abeb-7c58ddb30139-kube-api-access-67vx9\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.856519 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.856498    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhsvw\" (UniqueName: \"kubernetes.io/projected/77090b45-e309-490a-973e-28c723351e73-kube-api-access-mhsvw\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:29.857029 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.857009    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hcdc\" (UniqueName: \"kubernetes.io/projected/a4f29e73-17e3-42cd-8bb5-980dfe3729c2-kube-api-access-9hcdc\") pod \"node-resolver-6gzn7\" (UID: \"a4f29e73-17e3-42cd-8bb5-980dfe3729c2\") " pod="openshift-dns/node-resolver-6gzn7"
Jul 08 06:54:29.857329 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.857309    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ss7n\" (UniqueName: \"kubernetes.io/projected/5b951e1f-2e78-4d1f-af51-eedaabc4a02a-kube-api-access-4ss7n\") pod \"iptables-alerter-xj4hz\" (UID: \"5b951e1f-2e78-4d1f-af51-eedaabc4a02a\") " pod="openshift-network-operator/iptables-alerter-xj4hz"
Jul 08 06:54:29.858548 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.858530    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ztl5\" (UniqueName: \"kubernetes.io/projected/c3f86a51-e751-4c0d-9958-611e61cdf7b0-kube-api-access-8ztl5\") pod \"ovnkube-node-8wkwl\" (UID: \"c3f86a51-e751-4c0d-9958-611e61cdf7b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.859495 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.859478    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndp7j\" (UniqueName: \"kubernetes.io/projected/d6e67cb8-a56a-4787-b0cb-a95cab8a760a-kube-api-access-ndp7j\") pod \"multus-9lcnh\" (UID: \"d6e67cb8-a56a-4787-b0cb-a95cab8a760a\") " pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:29.863023 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.863004    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d573e55-4edc-47fb-abeb-7c58ddb30139-tmp\") pod \"tuned-stt62\" (UID: \"0d573e55-4edc-47fb-abeb-7c58ddb30139\") " pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:29.942285 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.942263    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m5n5f"
Jul 08 06:54:29.951084 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.951063    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw"
Jul 08 06:54:29.960578 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.960562    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6gzn7"
Jul 08 06:54:29.967156 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.967138    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d5qzh"
Jul 08 06:54:29.973657 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.973637    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:29.981488 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.981466    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-crwf7"
Jul 08 06:54:29.989131 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.989115    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xj4hz"
Jul 08 06:54:29.995813 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:29.995796    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-stt62"
Jul 08 06:54:30.001731 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.001714    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9lcnh"
Jul 08 06:54:30.247620 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.247594    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:30.247892 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:30.247744    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:30.247892 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:30.247832    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret podName:39d02a2c-8802-46ce-a765-576cca29608e nodeName:}" failed. No retries permitted until 2026-07-08 06:54:31.247811637 +0000 UTC m=+3.984156352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret") pod "global-pull-secret-syncer-7zf27" (UID: "39d02a2c-8802-46ce-a765-576cca29608e") : object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:30.348995 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.348960    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:30.349159 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.349039    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9l4f\" (UniqueName: \"kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f\") pod \"network-check-target-h4bmr\" (UID: \"8de691b9-d68b-4ae5-a0aa-47587f146d7e\") " pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:30.349159 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:30.349123    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:30.349276 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:30.349182    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 06:54:30.349276 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:30.349202    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 06:54:30.349276 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:30.349215    2557 projected.go:196] Error preparing data for projected volume kube-api-access-x9l4f for pod openshift-network-diagnostics/network-check-target-h4bmr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:30.349276 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:30.349218    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs podName:77090b45-e309-490a-973e-28c723351e73 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:31.349198531 +0000 UTC m=+4.085543238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs") pod "network-metrics-daemon-r4hnb" (UID: "77090b45-e309-490a-973e-28c723351e73") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:30.349276 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:30.349266    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f podName:8de691b9-d68b-4ae5-a0aa-47587f146d7e nodeName:}" failed. No retries permitted until 2026-07-08 06:54:31.349249937 +0000 UTC m=+4.085594645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9l4f" (UniqueName: "kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f") pod "network-check-target-h4bmr" (UID: "8de691b9-d68b-4ae5-a0aa-47587f146d7e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:30.623286 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:30.623256    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b951e1f_2e78_4d1f_af51_eedaabc4a02a.slice/crio-50cac04cc9b2eb26d0ec7fdfcb6961985f5d7469e64c5c85797fd2c87a4391af WatchSource:0}: Error finding container 50cac04cc9b2eb26d0ec7fdfcb6961985f5d7469e64c5c85797fd2c87a4391af: Status 404 returned error can't find the container with id 50cac04cc9b2eb26d0ec7fdfcb6961985f5d7469e64c5c85797fd2c87a4391af
Jul 08 06:54:30.624996 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:30.624957    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f86a51_e751_4c0d_9958_611e61cdf7b0.slice/crio-035ac7e607989e2565f5355b33539ca2167f474a37f43c6cb629dd7d848c31ca WatchSource:0}: Error finding container 035ac7e607989e2565f5355b33539ca2167f474a37f43c6cb629dd7d848c31ca: Status 404 returned error can't find the container with id 035ac7e607989e2565f5355b33539ca2167f474a37f43c6cb629dd7d848c31ca
Jul 08 06:54:30.628037 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:30.628011    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56214b24_626b_4361_965b_b6b26940ac5d.slice/crio-02169164790e70e44301660c0e50aa0b3746cec0e52961cb65620b2124b1a527 WatchSource:0}: Error finding container 02169164790e70e44301660c0e50aa0b3746cec0e52961cb65620b2124b1a527: Status 404 returned error can't find the container with id 02169164790e70e44301660c0e50aa0b3746cec0e52961cb65620b2124b1a527
Jul 08 06:54:30.629009 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:30.628896    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b35714_946f_4024_8c67_ca8e4994f3a2.slice/crio-610b00d1e085f1c13e71817b2c97105784c50517e42e03528f956401bb2c4c58 WatchSource:0}: Error finding container 610b00d1e085f1c13e71817b2c97105784c50517e42e03528f956401bb2c4c58: Status 404 returned error can't find the container with id 610b00d1e085f1c13e71817b2c97105784c50517e42e03528f956401bb2c4c58
Jul 08 06:54:30.635186 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:54:30.635123    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa12ff9_ed84_4ecc_b80e_ee6fd82d91a6.slice/crio-2ec9a7c7caaccf3ff11ad62ea753ed2ae6e554fa11bf9a055cc7caee32144cd1 WatchSource:0}: Error finding container 2ec9a7c7caaccf3ff11ad62ea753ed2ae6e554fa11bf9a055cc7caee32144cd1: Status 404 returned error can't find the container with id 2ec9a7c7caaccf3ff11ad62ea753ed2ae6e554fa11bf9a055cc7caee32144cd1
Jul 08 06:54:30.681093 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.680957    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 06:49:29 +0000 UTC" deadline="2028-01-29 19:52:10.770745631 +0000 UTC"
Jul 08 06:54:30.681179 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.681095    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13692h57m40.089655583s"
Jul 08 06:54:30.780105 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.780082    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:30.780235 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:30.780181    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:30.790268 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.790237    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-stt62" event={"ID":"0d573e55-4edc-47fb-abeb-7c58ddb30139","Type":"ContainerStarted","Data":"47c0d56f1bb43f9b0c69c0f011c791600cb095067b6ca43a1335efc4e2c925e0"}
Jul 08 06:54:30.792513 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.792486    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m5n5f" event={"ID":"e3b35714-946f-4024-8c67-ca8e4994f3a2","Type":"ContainerStarted","Data":"610b00d1e085f1c13e71817b2c97105784c50517e42e03528f956401bb2c4c58"}
Jul 08 06:54:30.794801 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.794777    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl" event={"ID":"c3f86a51-e751-4c0d-9958-611e61cdf7b0","Type":"ContainerStarted","Data":"035ac7e607989e2565f5355b33539ca2167f474a37f43c6cb629dd7d848c31ca"}
Jul 08 06:54:30.795728 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.795706    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xj4hz" event={"ID":"5b951e1f-2e78-4d1f-af51-eedaabc4a02a","Type":"ContainerStarted","Data":"50cac04cc9b2eb26d0ec7fdfcb6961985f5d7469e64c5c85797fd2c87a4391af"}
Jul 08 06:54:30.797574 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.797552    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-211.ec2.internal" event={"ID":"bea9ecd901e1fb22501c8126e0da3e1b","Type":"ContainerStarted","Data":"a6601f5db35ce9e36c7e9f53284652632778dc528e20c88f62fc5030ed4be2d4"}
Jul 08 06:54:30.798601 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.798549    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw" event={"ID":"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6","Type":"ContainerStarted","Data":"2ec9a7c7caaccf3ff11ad62ea753ed2ae6e554fa11bf9a055cc7caee32144cd1"}
Jul 08 06:54:30.800212 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.800176    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9lcnh" event={"ID":"d6e67cb8-a56a-4787-b0cb-a95cab8a760a","Type":"ContainerStarted","Data":"dfa9f4897a2ed3ab0cecb917eefb4257499d158c3200a53d27f1cb7546ec3497"}
Jul 08 06:54:30.801144 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.801126    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6gzn7" event={"ID":"a4f29e73-17e3-42cd-8bb5-980dfe3729c2","Type":"ContainerStarted","Data":"ccf85124334b50cbe0f5a5e79b60b36c765241ec64229d5f400f4f6192833559"}
Jul 08 06:54:30.802096 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.802069    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d5qzh" event={"ID":"fd567957-630a-4c89-8890-7abcf95e7298","Type":"ContainerStarted","Data":"41f2d2392e6df7d09f16606e97dfe1ea3d20e9d1f609af3385fdb1f8f799e7ba"}
Jul 08 06:54:30.804009 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.803990    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crwf7" event={"ID":"56214b24-626b-4361-965b-b6b26940ac5d","Type":"ContainerStarted","Data":"02169164790e70e44301660c0e50aa0b3746cec0e52961cb65620b2124b1a527"}
Jul 08 06:54:30.812950 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:30.812909    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-211.ec2.internal" podStartSLOduration=2.812898155 podStartE2EDuration="2.812898155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:54:30.812748109 +0000 UTC m=+3.549092820" watchObservedRunningTime="2026-07-08 06:54:30.812898155 +0000 UTC m=+3.549242869"
Jul 08 06:54:31.257898 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:31.257827    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:31.258092 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:31.258025    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:31.258167 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:31.258105    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret podName:39d02a2c-8802-46ce-a765-576cca29608e nodeName:}" failed. No retries permitted until 2026-07-08 06:54:33.258086299 +0000 UTC m=+5.994430991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret") pod "global-pull-secret-syncer-7zf27" (UID: "39d02a2c-8802-46ce-a765-576cca29608e") : object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:31.358376 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:31.358324    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9l4f\" (UniqueName: \"kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f\") pod \"network-check-target-h4bmr\" (UID: \"8de691b9-d68b-4ae5-a0aa-47587f146d7e\") " pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:31.358537 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:31.358411    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:31.358537 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:31.358534    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:31.358659 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:31.358536    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 06:54:31.358659 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:31.358559    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 06:54:31.358659 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:31.358571    2557 projected.go:196] Error preparing data for projected volume kube-api-access-x9l4f for pod openshift-network-diagnostics/network-check-target-h4bmr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:31.358659 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:31.358590    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs podName:77090b45-e309-490a-973e-28c723351e73 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:33.358572379 +0000 UTC m=+6.094917074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs") pod "network-metrics-daemon-r4hnb" (UID: "77090b45-e309-490a-973e-28c723351e73") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:31.358659 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:31.358604    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f podName:8de691b9-d68b-4ae5-a0aa-47587f146d7e nodeName:}" failed. No retries permitted until 2026-07-08 06:54:33.358595015 +0000 UTC m=+6.094939724 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9l4f" (UniqueName: "kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f") pod "network-check-target-h4bmr" (UID: "8de691b9-d68b-4ae5-a0aa-47587f146d7e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:31.681841 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:31.681799    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 06:49:29 +0000 UTC" deadline="2027-12-22 11:29:55.181332026 +0000 UTC"
Jul 08 06:54:31.681841 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:31.681839    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12772h35m23.499497776s"
Jul 08 06:54:31.784160 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:31.783248    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:31.784160 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:31.783395    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:31.784160 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:31.783902    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:31.784160 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:31.783993    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:31.830909 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:31.829632    2557 generic.go:356] "Generic (PLEG): container finished" podID="39302b21a7247f89d983575aca27ff92" containerID="da86caf775eda771a506580d039d7eb2c93736beec9864538d99dcc8bfdede2c" exitCode=0
Jul 08 06:54:31.830909 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:31.830589    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal" event={"ID":"39302b21a7247f89d983575aca27ff92","Type":"ContainerDied","Data":"da86caf775eda771a506580d039d7eb2c93736beec9864538d99dcc8bfdede2c"}
Jul 08 06:54:32.780642 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:32.780612    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:32.781148 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:32.780745    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:32.843884 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:32.843841    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal" event={"ID":"39302b21a7247f89d983575aca27ff92","Type":"ContainerStarted","Data":"0fefe8c66af268d4a7f298484384356bb4c2d69e8a48962b505ab49d04cc91ca"}
Jul 08 06:54:33.275027 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:33.274988    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:33.275292 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:33.275173    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:33.275292 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:33.275234    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret podName:39d02a2c-8802-46ce-a765-576cca29608e nodeName:}" failed. No retries permitted until 2026-07-08 06:54:37.27521586 +0000 UTC m=+10.011560581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret") pod "global-pull-secret-syncer-7zf27" (UID: "39d02a2c-8802-46ce-a765-576cca29608e") : object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:33.376964 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:33.376158    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9l4f\" (UniqueName: \"kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f\") pod \"network-check-target-h4bmr\" (UID: \"8de691b9-d68b-4ae5-a0aa-47587f146d7e\") " pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:33.376964 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:33.376220    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:33.376964 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:33.376363    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:33.376964 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:33.376422    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs podName:77090b45-e309-490a-973e-28c723351e73 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:37.376404348 +0000 UTC m=+10.112749045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs") pod "network-metrics-daemon-r4hnb" (UID: "77090b45-e309-490a-973e-28c723351e73") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:33.376964 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:33.376856    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 06:54:33.376964 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:33.376877    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 06:54:33.376964 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:33.376890    2557 projected.go:196] Error preparing data for projected volume kube-api-access-x9l4f for pod openshift-network-diagnostics/network-check-target-h4bmr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:33.376964 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:33.376933    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f podName:8de691b9-d68b-4ae5-a0aa-47587f146d7e nodeName:}" failed. No retries permitted until 2026-07-08 06:54:37.376918453 +0000 UTC m=+10.113263150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9l4f" (UniqueName: "kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f") pod "network-check-target-h4bmr" (UID: "8de691b9-d68b-4ae5-a0aa-47587f146d7e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:33.780875 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:33.780311    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:33.780875 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:33.780433    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:33.780875 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:33.780490    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:33.780875 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:33.780622    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:34.780624 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:34.780594    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:34.780815 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:34.780718    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:35.781931 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:35.781903    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:35.782427 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:35.782012    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:35.782490 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:35.782453    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:35.782585 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:35.782563    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:36.780248 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:36.780220    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:36.780428 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:36.780328    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:37.310742 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:37.310707    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:37.311193 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:37.310871    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:37.311193 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:37.310944    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret podName:39d02a2c-8802-46ce-a765-576cca29608e nodeName:}" failed. No retries permitted until 2026-07-08 06:54:45.310924871 +0000 UTC m=+18.047269577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret") pod "global-pull-secret-syncer-7zf27" (UID: "39d02a2c-8802-46ce-a765-576cca29608e") : object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:37.412271 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:37.411500    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9l4f\" (UniqueName: \"kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f\") pod \"network-check-target-h4bmr\" (UID: \"8de691b9-d68b-4ae5-a0aa-47587f146d7e\") " pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:37.412271 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:37.411557    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:37.412271 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:37.411687    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:37.412271 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:37.411745    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs podName:77090b45-e309-490a-973e-28c723351e73 nodeName:}" failed. No retries permitted until 2026-07-08 06:54:45.411726149 +0000 UTC m=+18.148070867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs") pod "network-metrics-daemon-r4hnb" (UID: "77090b45-e309-490a-973e-28c723351e73") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:37.412271 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:37.412159    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 06:54:37.412271 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:37.412177    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 06:54:37.412271 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:37.412192    2557 projected.go:196] Error preparing data for projected volume kube-api-access-x9l4f for pod openshift-network-diagnostics/network-check-target-h4bmr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:37.412271 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:37.412238    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f podName:8de691b9-d68b-4ae5-a0aa-47587f146d7e nodeName:}" failed. No retries permitted until 2026-07-08 06:54:45.412222518 +0000 UTC m=+18.148567216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9l4f" (UniqueName: "kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f") pod "network-check-target-h4bmr" (UID: "8de691b9-d68b-4ae5-a0aa-47587f146d7e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:37.781167 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:37.780651    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:37.781167 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:37.780784    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:37.781167 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:37.781065    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:37.781167 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:37.781134    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:38.780640 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:38.780610    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:38.781080 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:38.780733    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:39.780316 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:39.780250    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:39.780316 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:39.780282    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:39.780513 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:39.780404    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:39.780655 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:39.780633    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:40.780001 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:40.779949    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:40.780423 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:40.780067    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:41.780423 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:41.779966    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:41.780423 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:41.780087    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:41.780423 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:41.780151    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:41.780423 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:41.780262    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:42.780446 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:42.780413    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:42.780927 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:42.780543    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:43.780154 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:43.780118    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:43.780308 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:43.780236    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:43.780353 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:43.780299    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:43.780441 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:43.780420    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:44.780325 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:44.780296    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:44.780832 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:44.780422    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:45.372434 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:45.372400    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:45.372661 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:45.372519    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:45.372661 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:45.372585    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret podName:39d02a2c-8802-46ce-a765-576cca29608e nodeName:}" failed. No retries permitted until 2026-07-08 06:55:01.37256779 +0000 UTC m=+34.108912482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret") pod "global-pull-secret-syncer-7zf27" (UID: "39d02a2c-8802-46ce-a765-576cca29608e") : object "kube-system"/"original-pull-secret" not registered
Jul 08 06:54:45.473635 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:45.473601    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:45.473829 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:45.473678    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9l4f\" (UniqueName: \"kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f\") pod \"network-check-target-h4bmr\" (UID: \"8de691b9-d68b-4ae5-a0aa-47587f146d7e\") " pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:45.473829 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:45.473782    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:45.473945 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:45.473832    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 06:54:45.473945 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:45.473848    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 06:54:45.473945 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:45.473853    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs podName:77090b45-e309-490a-973e-28c723351e73 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:01.473832359 +0000 UTC m=+34.210177053 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs") pod "network-metrics-daemon-r4hnb" (UID: "77090b45-e309-490a-973e-28c723351e73") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:54:45.473945 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:45.473861    2557 projected.go:196] Error preparing data for projected volume kube-api-access-x9l4f for pod openshift-network-diagnostics/network-check-target-h4bmr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:45.473945 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:45.473911    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f podName:8de691b9-d68b-4ae5-a0aa-47587f146d7e nodeName:}" failed. No retries permitted until 2026-07-08 06:55:01.473895273 +0000 UTC m=+34.210239987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9l4f" (UniqueName: "kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f") pod "network-check-target-h4bmr" (UID: "8de691b9-d68b-4ae5-a0aa-47587f146d7e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:54:45.781789 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:45.780333    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:45.781789 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:45.780458    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:45.781789 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:45.780467    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:45.781789 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:45.780569    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:46.780709 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:46.780671    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:46.780923 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:46.780813    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:47.781462 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:47.781425    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:47.781915 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:47.781534    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:47.781915 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:47.781566    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:47.781915 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:47.781657    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:48.779723 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:48.779693    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:48.779915 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:48.779834    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:49.780584 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:49.780551    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:49.780973 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:49.780649    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:49.780973 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:49.780774    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:49.780973 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:49.780896    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:50.780295 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.779982    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:50.780457 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:50.780407    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:50.881076 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.880999    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-stt62" event={"ID":"0d573e55-4edc-47fb-abeb-7c58ddb30139","Type":"ContainerStarted","Data":"0457eea4e134f5ed788cc51e193b0c488df84be75d0231953459935d225f96e1"}
Jul 08 06:54:50.883068 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.883042    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m5n5f" event={"ID":"e3b35714-946f-4024-8c67-ca8e4994f3a2","Type":"ContainerStarted","Data":"329e4a8b6a9d1aaa71529f3cc721add89baa8c35bebaff5c2339c98c87e758db"}
Jul 08 06:54:50.886349 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.886322    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl" event={"ID":"c3f86a51-e751-4c0d-9958-611e61cdf7b0","Type":"ContainerStarted","Data":"198f737aaa9e8fa0804cd40f0f008086542eac0e33d80d5cff33cb20d1159103"}
Jul 08 06:54:50.886446 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.886395    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl" event={"ID":"c3f86a51-e751-4c0d-9958-611e61cdf7b0","Type":"ContainerStarted","Data":"7662faa7d07d1d742bddfd2c591e0963b4bdaf4a5af16ca1860dcf913e1f3571"}
Jul 08 06:54:50.886512 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.886447    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl" event={"ID":"c3f86a51-e751-4c0d-9958-611e61cdf7b0","Type":"ContainerStarted","Data":"6261665d5753b5835e99a75aff08c38c4fd03785c750a0d435970b0d3d64e871"}
Jul 08 06:54:50.886512 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.886493    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl" event={"ID":"c3f86a51-e751-4c0d-9958-611e61cdf7b0","Type":"ContainerStarted","Data":"21fc85f165094d3d7cc86b35765d3551267bd8bc17c2a489d64003b5e30bf7b9"}
Jul 08 06:54:50.886614 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.886540    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl" event={"ID":"c3f86a51-e751-4c0d-9958-611e61cdf7b0","Type":"ContainerStarted","Data":"00f6ecea159c0d9e4eac532471511d34358557619df64ce566123e02130e0204"}
Jul 08 06:54:50.886614 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.886589    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl" event={"ID":"c3f86a51-e751-4c0d-9958-611e61cdf7b0","Type":"ContainerStarted","Data":"6e48a0997a99886f1f98294eed957cc33d610c111499cece2c4935355e6224ea"}
Jul 08 06:54:50.887984 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.887951    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw" event={"ID":"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6","Type":"ContainerStarted","Data":"b752b6944cc2562eadfff83518de68f9d7f03b5f8c322c4e5b6190c876a9d76f"}
Jul 08 06:54:50.889388 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.889365    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9lcnh" event={"ID":"d6e67cb8-a56a-4787-b0cb-a95cab8a760a","Type":"ContainerStarted","Data":"472f0811b7d956922bd1fd9b062bfdcac587b26f1ff24ea404df3a29d820f5df"}
Jul 08 06:54:50.891189 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.891160    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6gzn7" event={"ID":"a4f29e73-17e3-42cd-8bb5-980dfe3729c2","Type":"ContainerStarted","Data":"70b1faf6cc48f8c96b16fe178dcb49a561b49bff9f8044f9f3bdf432020e92b9"}
Jul 08 06:54:50.892677 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.892657    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d5qzh" event={"ID":"fd567957-630a-4c89-8890-7abcf95e7298","Type":"ContainerStarted","Data":"2a58d21ebcae4144490e7e06670b38bf11908b4cb1cb29bc52d1744a295a36a1"}
Jul 08 06:54:50.894029 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.894007    2557 generic.go:356] "Generic (PLEG): container finished" podID="56214b24-626b-4361-965b-b6b26940ac5d" containerID="b4d43e8dfb514c83b0d0c33776819c08d08cac197641751479fd4ee1769dfb96" exitCode=0
Jul 08 06:54:50.894115 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.894039    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crwf7" event={"ID":"56214b24-626b-4361-965b-b6b26940ac5d","Type":"ContainerDied","Data":"b4d43e8dfb514c83b0d0c33776819c08d08cac197641751479fd4ee1769dfb96"}
Jul 08 06:54:50.897496 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.897457    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-211.ec2.internal" podStartSLOduration=22.89744634 podStartE2EDuration="22.89744634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:54:32.85919409 +0000 UTC m=+5.595538805" watchObservedRunningTime="2026-07-08 06:54:50.89744634 +0000 UTC m=+23.633791054"
Jul 08 06:54:50.897682 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.897653    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-stt62" podStartSLOduration=4.471568782 podStartE2EDuration="23.897645316s" totalImagesPullingTime="19.426076534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:54:50.89736074 +0000 UTC m=+23.633705454" watchObservedRunningTime="2026-07-08 06:54:50.897645316 +0000 UTC m=+23.633990030"
Jul 08 06:54:50.910965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.910932    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6gzn7" podStartSLOduration=4.47890699 podStartE2EDuration="23.910920285s" totalImagesPullingTime="19.432013295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:54:50.910533168 +0000 UTC m=+23.646877881" watchObservedRunningTime="2026-07-08 06:54:50.910920285 +0000 UTC m=+23.647264999"
Jul 08 06:54:50.936065 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.936033    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9lcnh" podStartSLOduration=4.509419838 podStartE2EDuration="23.936026168s" totalImagesPullingTime="19.42660633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:54:50.935882419 +0000 UTC m=+23.672227132" watchObservedRunningTime="2026-07-08 06:54:50.936026168 +0000 UTC m=+23.672370878"
Jul 08 06:54:50.957461 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.957425    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m5n5f" podStartSLOduration=4.601033336 podStartE2EDuration="23.957413794s" totalImagesPullingTime="19.356380458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:54:50.957143179 +0000 UTC m=+23.693487904" watchObservedRunningTime="2026-07-08 06:54:50.957413794 +0000 UTC m=+23.693758508"
Jul 08 06:54:50.976836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:50.976796    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d5qzh" podStartSLOduration=4.57422017 podStartE2EDuration="23.976780945s" totalImagesPullingTime="19.402560775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:54:50.976377557 +0000 UTC m=+23.712722271" watchObservedRunningTime="2026-07-08 06:54:50.976780945 +0000 UTC m=+23.713125661"
Jul 08 06:54:51.516946 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:51.516925    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 08 06:54:51.707275 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:51.707128    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T06:54:51.516940824Z","UUID":"402188df-feaf-4513-8ce2-ef844f46b9ba","Handler":null,"Name":"","Endpoint":""}
Jul 08 06:54:51.708681 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:51.708661    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 08 06:54:51.708822 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:51.708701    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 08 06:54:51.779917 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:51.779850    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:51.779917 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:51.779893    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:51.780146 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:51.780006    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:51.780207 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:51.780146    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:51.897910 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:51.897871    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xj4hz" event={"ID":"5b951e1f-2e78-4d1f-af51-eedaabc4a02a","Type":"ContainerStarted","Data":"524012d6ddd28df0c3a0ece49670f66993aa2fca7bbd286ebc577aa7352efab8"}
Jul 08 06:54:51.900024 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:51.899993    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw" event={"ID":"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6","Type":"ContainerStarted","Data":"83d3278176aae91251a83af04acede3853db55f4d85fb5e7f640f8e4549e84b1"}
Jul 08 06:54:51.917519 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:51.917476    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xj4hz" podStartSLOduration=5.53146261 podStartE2EDuration="24.917455563s" totalImagesPullingTime="19.385992953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:54:51.917085934 +0000 UTC m=+24.653430648" watchObservedRunningTime="2026-07-08 06:54:51.917455563 +0000 UTC m=+24.653800279"
Jul 08 06:54:52.079809 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:52.079682    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m5n5f"
Jul 08 06:54:52.080435 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:52.080382    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m5n5f"
Jul 08 06:54:52.780704 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:52.780498    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:52.780866 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:52.780818    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:52.904851 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:52.904822    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw" event={"ID":"aaa12ff9-ed84-4ecc-b80e-ee6fd82d91a6","Type":"ContainerStarted","Data":"ae451cefb9d3496d270ebf3e74d78a932cadb1cd5b06c4d72a56521693bcde30"}
Jul 08 06:54:52.905188 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:52.905008    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m5n5f"
Jul 08 06:54:52.905443 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:52.905424    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m5n5f"
Jul 08 06:54:52.923578 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:52.923538    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7phvw" podStartSLOduration=4.415429709 podStartE2EDuration="25.923523697s" totalImagesPullingTime="21.508093988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:54:52.922826681 +0000 UTC m=+25.659171395" watchObservedRunningTime="2026-07-08 06:54:52.923523697 +0000 UTC m=+25.659868444"
Jul 08 06:54:53.780250 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:53.780221    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:53.780440 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:53.780314    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:53.780440 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:53.780418    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:53.780582 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:53.780560    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:53.910899 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:53.910859    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl" event={"ID":"c3f86a51-e751-4c0d-9958-611e61cdf7b0","Type":"ContainerStarted","Data":"dee22d305275ae23fa94a96a272c60bd8007ddc13c1f90e6ac9d02f307d21406"}
Jul 08 06:54:54.780157 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:54.780130    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:54.780344 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:54.780231    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:55.780373 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:55.780200    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:55.781089 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:55.780269    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:55.781089 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:55.780453    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:55.781089 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:55.780531    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:55.916553 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:55.916493    2557 generic.go:356] "Generic (PLEG): container finished" podID="56214b24-626b-4361-965b-b6b26940ac5d" containerID="ceb38c528c57a46e5e28a47dd36cc4ba597f9455bfe8c930075e943943730e8c" exitCode=0
Jul 08 06:54:55.916684 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:55.916573    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crwf7" event={"ID":"56214b24-626b-4361-965b-b6b26940ac5d","Type":"ContainerDied","Data":"ceb38c528c57a46e5e28a47dd36cc4ba597f9455bfe8c930075e943943730e8c"}
Jul 08 06:54:55.920332 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:55.920307    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl" event={"ID":"c3f86a51-e751-4c0d-9958-611e61cdf7b0","Type":"ContainerStarted","Data":"28c25da2c5623db1d7f84059adc1a015f33cb1992517cf2c1fd139359743cc05"}
Jul 08 06:54:55.920525 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:55.920513    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:55.920597 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:55.920553    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:55.936410 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:55.936386    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:55.957325 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:55.957272    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl" podStartSLOduration=9.374143646 podStartE2EDuration="28.957260242s" totalImagesPullingTime="19.583116596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:54:55.956383889 +0000 UTC m=+28.692728604" watchObservedRunningTime="2026-07-08 06:54:55.957260242 +0000 UTC m=+28.693604953"
Jul 08 06:54:56.780226 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:56.780197    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:56.780401 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:56.780300    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:56.922992 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:56.922949    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:56.938011 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:56.937986    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:54:57.417106 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:57.416900    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4bmr"]
Jul 08 06:54:57.417306 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:57.417190    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:57.417306 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:57.417292    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:57.420232 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:57.420191    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r4hnb"]
Jul 08 06:54:57.420361 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:57.420308    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:57.420443 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:57.420417    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:57.420902 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:57.420884    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7zf27"]
Jul 08 06:54:57.420994 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:57.420981    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:57.421073 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:57.421056    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:57.925981 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:57.925947    2557 generic.go:356] "Generic (PLEG): container finished" podID="56214b24-626b-4361-965b-b6b26940ac5d" containerID="a593f6777bc19e769808a38661a1f7d44c0740235af02de0364be88624d54ca9" exitCode=0
Jul 08 06:54:57.926445 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:57.926029    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crwf7" event={"ID":"56214b24-626b-4361-965b-b6b26940ac5d","Type":"ContainerDied","Data":"a593f6777bc19e769808a38661a1f7d44c0740235af02de0364be88624d54ca9"}
Jul 08 06:54:58.780207 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:58.780170    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:54:58.780207 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:58.780208    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:54:58.780444 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:58.780309    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:54:58.780444 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:58.780422    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:54:59.780451 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:59.780381    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:54:59.781098 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:54:59.780486    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:54:59.931272 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:59.931245    2557 generic.go:356] "Generic (PLEG): container finished" podID="56214b24-626b-4361-965b-b6b26940ac5d" containerID="534ad2bdcdf83ce9c904186940cf9c8e0ec5337f3e79c78ff3d813ab6f7f6cbd" exitCode=0
Jul 08 06:54:59.931402 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:54:59.931302    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crwf7" event={"ID":"56214b24-626b-4361-965b-b6b26940ac5d","Type":"ContainerDied","Data":"534ad2bdcdf83ce9c904186940cf9c8e0ec5337f3e79c78ff3d813ab6f7f6cbd"}
Jul 08 06:55:00.780072 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:00.780042    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:55:00.780234 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:00.780048    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:55:00.780234 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:00.780138    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:55:00.780353 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:00.780237    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:55:01.393825 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:01.393793    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:55:01.394639 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:01.393957    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 06:55:01.394639 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:01.394033    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret podName:39d02a2c-8802-46ce-a765-576cca29608e nodeName:}" failed. No retries permitted until 2026-07-08 06:55:33.394011624 +0000 UTC m=+66.130356317 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret") pod "global-pull-secret-syncer-7zf27" (UID: "39d02a2c-8802-46ce-a765-576cca29608e") : object "kube-system"/"original-pull-secret" not registered
Jul 08 06:55:01.494437 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:01.494410    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9l4f\" (UniqueName: \"kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f\") pod \"network-check-target-h4bmr\" (UID: \"8de691b9-d68b-4ae5-a0aa-47587f146d7e\") " pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:55:01.494598 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:01.494461    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:55:01.494598 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:01.494567    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 06:55:01.494598 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:01.494589    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 06:55:01.494728 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:01.494603    2557 projected.go:196] Error preparing data for projected volume kube-api-access-x9l4f for pod openshift-network-diagnostics/network-check-target-h4bmr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:55:01.494728 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:01.494607    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:55:01.494728 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:01.494662    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f podName:8de691b9-d68b-4ae5-a0aa-47587f146d7e nodeName:}" failed. No retries permitted until 2026-07-08 06:55:33.494643251 +0000 UTC m=+66.230987947 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x9l4f" (UniqueName: "kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f") pod "network-check-target-h4bmr" (UID: "8de691b9-d68b-4ae5-a0aa-47587f146d7e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 06:55:01.494728 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:01.494681    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs podName:77090b45-e309-490a-973e-28c723351e73 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:33.494671407 +0000 UTC m=+66.231016108 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs") pod "network-metrics-daemon-r4hnb" (UID: "77090b45-e309-490a-973e-28c723351e73") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 06:55:01.780449 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:01.780418    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:55:01.780617 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:01.780536    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4bmr" podUID="8de691b9-d68b-4ae5-a0aa-47587f146d7e"
Jul 08 06:55:02.780701 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:02.780670    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:55:02.781163 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:02.780714    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:55:02.781163 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:02.780820    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7zf27" podUID="39d02a2c-8802-46ce-a765-576cca29608e"
Jul 08 06:55:02.781163 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:02.780943    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:55:03.005106 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.005080    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-211.ec2.internal" event="NodeReady"
Jul 08 06:55:03.005293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.005204    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 08 06:55:03.052727 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.052698    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gtnj4"]
Jul 08 06:55:03.056324 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.056303    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lcn7q"]
Jul 08 06:55:03.056453 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.056433    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:03.058885 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.058863    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 08 06:55:03.059027 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.058914    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vxwlb\""
Jul 08 06:55:03.059094 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.059086    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 08 06:55:03.059446 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.059429    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lcn7q"
Jul 08 06:55:03.061909 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.061506    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 08 06:55:03.061909 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.061610    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 08 06:55:03.061909 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.061652    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 08 06:55:03.061909 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.061781    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2tvxw\""
Jul 08 06:55:03.066928 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.066907    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gtnj4"]
Jul 08 06:55:03.068456 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.068415    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lcn7q"]
Jul 08 06:55:03.207770 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.207738    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/031afabe-d529-4539-b152-d45a7c651d42-config-volume\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:03.207770 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.207783    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:03.207967 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.207807    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4847909b-7f1a-44f2-9f42-d017286c2d6d-cert\") pod \"ingress-canary-lcn7q\" (UID: \"4847909b-7f1a-44f2-9f42-d017286c2d6d\") " pod="openshift-ingress-canary/ingress-canary-lcn7q"
Jul 08 06:55:03.207967 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.207892    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031afabe-d529-4539-b152-d45a7c651d42-tmp-dir\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:03.207967 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.207944    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nttq6\" (UniqueName: \"kubernetes.io/projected/031afabe-d529-4539-b152-d45a7c651d42-kube-api-access-nttq6\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:03.208139 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.207970    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z295k\" (UniqueName: \"kubernetes.io/projected/4847909b-7f1a-44f2-9f42-d017286c2d6d-kube-api-access-z295k\") pod \"ingress-canary-lcn7q\" (UID: \"4847909b-7f1a-44f2-9f42-d017286c2d6d\") " pod="openshift-ingress-canary/ingress-canary-lcn7q"
Jul 08 06:55:03.309127 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.309042    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031afabe-d529-4539-b152-d45a7c651d42-tmp-dir\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:03.309127 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.309122    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nttq6\" (UniqueName: \"kubernetes.io/projected/031afabe-d529-4539-b152-d45a7c651d42-kube-api-access-nttq6\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:03.309348 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.309150    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z295k\" (UniqueName: \"kubernetes.io/projected/4847909b-7f1a-44f2-9f42-d017286c2d6d-kube-api-access-z295k\") pod \"ingress-canary-lcn7q\" (UID: \"4847909b-7f1a-44f2-9f42-d017286c2d6d\") " pod="openshift-ingress-canary/ingress-canary-lcn7q"
Jul 08 06:55:03.309348 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.309208    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/031afabe-d529-4539-b152-d45a7c651d42-config-volume\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:03.309348 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.309238    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:03.309348 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.309260    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4847909b-7f1a-44f2-9f42-d017286c2d6d-cert\") pod \"ingress-canary-lcn7q\" (UID: \"4847909b-7f1a-44f2-9f42-d017286c2d6d\") " pod="openshift-ingress-canary/ingress-canary-lcn7q"
Jul 08 06:55:03.309529 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:03.309378    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:55:03.309529 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.309406    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031afabe-d529-4539-b152-d45a7c651d42-tmp-dir\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:03.309529 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:03.309468    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls podName:031afabe-d529-4539-b152-d45a7c651d42 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:03.809450401 +0000 UTC m=+36.545795109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls") pod "dns-default-gtnj4" (UID: "031afabe-d529-4539-b152-d45a7c651d42") : secret "dns-default-metrics-tls" not found
Jul 08 06:55:03.310013 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.309815    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/031afabe-d529-4539-b152-d45a7c651d42-config-volume\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:03.314978 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.314951    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4847909b-7f1a-44f2-9f42-d017286c2d6d-cert\") pod \"ingress-canary-lcn7q\" (UID: \"4847909b-7f1a-44f2-9f42-d017286c2d6d\") " pod="openshift-ingress-canary/ingress-canary-lcn7q"
Jul 08 06:55:03.318397 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.318374    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nttq6\" (UniqueName: \"kubernetes.io/projected/031afabe-d529-4539-b152-d45a7c651d42-kube-api-access-nttq6\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:03.318905 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.318882    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z295k\" (UniqueName: \"kubernetes.io/projected/4847909b-7f1a-44f2-9f42-d017286c2d6d-kube-api-access-z295k\") pod \"ingress-canary-lcn7q\" (UID: \"4847909b-7f1a-44f2-9f42-d017286c2d6d\") " pod="openshift-ingress-canary/ingress-canary-lcn7q"
Jul 08 06:55:03.374983 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.374959    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lcn7q"
Jul 08 06:55:03.780036 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.780001    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:55:03.783092 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.783073    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 06:55:03.784157 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.784136    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 06:55:03.784375 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.784358    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-smjcg\""
Jul 08 06:55:03.812885 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:03.812860    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:03.813004 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:03.812982    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:55:03.813053 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:03.813028    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls podName:031afabe-d529-4539-b152-d45a7c651d42 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:04.813015753 +0000 UTC m=+37.549360445 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls") pod "dns-default-gtnj4" (UID: "031afabe-d529-4539-b152-d45a7c651d42") : secret "dns-default-metrics-tls" not found
Jul 08 06:55:04.460272 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:04.460249    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lcn7q"]
Jul 08 06:55:04.463362 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:55:04.463340    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4847909b_7f1a_44f2_9f42_d017286c2d6d.slice/crio-1eeb1faa9fbf921d22c8b29c112c159f84972b006ef6535e70a82d8baa78f90a WatchSource:0}: Error finding container 1eeb1faa9fbf921d22c8b29c112c159f84972b006ef6535e70a82d8baa78f90a: Status 404 returned error can't find the container with id 1eeb1faa9fbf921d22c8b29c112c159f84972b006ef6535e70a82d8baa78f90a
Jul 08 06:55:04.780345 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:04.780316    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:55:04.780345 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:04.780343    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:55:04.782965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:04.782944    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 06:55:04.782965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:04.782960    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 06:55:04.783902 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:04.783886    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6plfv\""
Jul 08 06:55:04.820822 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:04.820750    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:04.820886 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:04.820874    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:55:04.820939 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:04.820929    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls podName:031afabe-d529-4539-b152-d45a7c651d42 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:06.820916336 +0000 UTC m=+39.557261030 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls") pod "dns-default-gtnj4" (UID: "031afabe-d529-4539-b152-d45a7c651d42") : secret "dns-default-metrics-tls" not found
Jul 08 06:55:04.945520 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:04.945499    2557 generic.go:356] "Generic (PLEG): container finished" podID="56214b24-626b-4361-965b-b6b26940ac5d" containerID="ca32c9c4f2aa6621206d8294507c4b6b97edb5f129afa905f0a7c8def4919385" exitCode=0
Jul 08 06:55:04.945604 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:04.945579    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crwf7" event={"ID":"56214b24-626b-4361-965b-b6b26940ac5d","Type":"ContainerDied","Data":"ca32c9c4f2aa6621206d8294507c4b6b97edb5f129afa905f0a7c8def4919385"}
Jul 08 06:55:04.946682 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:04.946627    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lcn7q" event={"ID":"4847909b-7f1a-44f2-9f42-d017286c2d6d","Type":"ContainerStarted","Data":"1eeb1faa9fbf921d22c8b29c112c159f84972b006ef6535e70a82d8baa78f90a"}
Jul 08 06:55:05.953212 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:05.953181    2557 generic.go:356] "Generic (PLEG): container finished" podID="56214b24-626b-4361-965b-b6b26940ac5d" containerID="e861721588bcc6aee2fb38705c72441b165956f1c51eaede879d1ccd0e82932c" exitCode=0
Jul 08 06:55:05.953582 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:05.953223    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crwf7" event={"ID":"56214b24-626b-4361-965b-b6b26940ac5d","Type":"ContainerDied","Data":"e861721588bcc6aee2fb38705c72441b165956f1c51eaede879d1ccd0e82932c"}
Jul 08 06:55:06.838953 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:06.838929    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:06.839078 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:06.839061    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:55:06.839122 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:06.839115    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls podName:031afabe-d529-4539-b152-d45a7c651d42 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:10.839102177 +0000 UTC m=+43.575446868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls") pod "dns-default-gtnj4" (UID: "031afabe-d529-4539-b152-d45a7c651d42") : secret "dns-default-metrics-tls" not found
Jul 08 06:55:06.958287 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:06.958261    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-crwf7" event={"ID":"56214b24-626b-4361-965b-b6b26940ac5d","Type":"ContainerStarted","Data":"a6ca9846e987a6f23e60b189c3440251bde1e1599b3b3eac47a624ae02be8d24"}
Jul 08 06:55:06.981392 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:06.981354    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-crwf7" podStartSLOduration=9.324341865 podStartE2EDuration="39.981339348s" totalImagesPullingTime="30.656997483s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:06.979647257 +0000 UTC m=+39.715991973" watchObservedRunningTime="2026-07-08 06:55:06.981339348 +0000 UTC m=+39.717684064"
Jul 08 06:55:07.961129 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:07.961093    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lcn7q" event={"ID":"4847909b-7f1a-44f2-9f42-d017286c2d6d","Type":"ContainerStarted","Data":"9140f46930a0849ff2470864e40a4a5b7183dbe35b63a2acc2a9a83a814540da"}
Jul 08 06:55:07.979880 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:07.978602    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lcn7q" podStartSLOduration=2.344224816 podStartE2EDuration="4.978582104s" totalImagesPullingTime="2.634357288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:07.977622062 +0000 UTC m=+40.713966776" watchObservedRunningTime="2026-07-08 06:55:07.978582104 +0000 UTC m=+40.714926819"
Jul 08 06:55:10.866938 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:10.866903    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:10.867305 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:10.867050    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:55:10.867305 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:10.867112    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls podName:031afabe-d529-4539-b152-d45a7c651d42 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:18.867097582 +0000 UTC m=+51.603442273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls") pod "dns-default-gtnj4" (UID: "031afabe-d529-4539-b152-d45a7c651d42") : secret "dns-default-metrics-tls" not found
Jul 08 06:55:18.929162 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:18.929124    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:18.929578 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:18.929222    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:55:18.929578 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:18.929280    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls podName:031afabe-d529-4539-b152-d45a7c651d42 nodeName:}" failed. No retries permitted until 2026-07-08 06:55:34.929266639 +0000 UTC m=+67.665611332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls") pod "dns-default-gtnj4" (UID: "031afabe-d529-4539-b152-d45a7c651d42") : secret "dns-default-metrics-tls" not found
Jul 08 06:55:28.948894 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:28.948867    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8wkwl"
Jul 08 06:55:33.424902 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:33.424869    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:55:33.427440 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:33.427422    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 06:55:33.438843 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:33.438828    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/39d02a2c-8802-46ce-a765-576cca29608e-original-pull-secret\") pod \"global-pull-secret-syncer-7zf27\" (UID: \"39d02a2c-8802-46ce-a765-576cca29608e\") " pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:55:33.525257 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:33.525231    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9l4f\" (UniqueName: \"kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f\") pod \"network-check-target-h4bmr\" (UID: \"8de691b9-d68b-4ae5-a0aa-47587f146d7e\") " pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:55:33.525366 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:33.525273    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:55:33.527957 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:33.527938    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 06:55:33.528078 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:33.528059    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 06:55:33.535912 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:33.535893    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 06:55:33.536000 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:33.535947    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs podName:77090b45-e309-490a-973e-28c723351e73 nodeName:}" failed. No retries permitted until 2026-07-08 06:56:37.535932071 +0000 UTC m=+130.272276763 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs") pod "network-metrics-daemon-r4hnb" (UID: "77090b45-e309-490a-973e-28c723351e73") : secret "metrics-daemon-secret" not found
Jul 08 06:55:33.537949 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:33.537934    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 06:55:33.548206 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:33.548190    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9l4f\" (UniqueName: \"kubernetes.io/projected/8de691b9-d68b-4ae5-a0aa-47587f146d7e-kube-api-access-x9l4f\") pod \"network-check-target-h4bmr\" (UID: \"8de691b9-d68b-4ae5-a0aa-47587f146d7e\") " pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:55:33.594250 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:33.594234    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7zf27"
Jul 08 06:55:33.712842 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:33.712786    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7zf27"]
Jul 08 06:55:33.715848 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:55:33.715821    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d02a2c_8802_46ce_a765_576cca29608e.slice/crio-6d7b743479df15557bc9fd212a53eee26b2c3d3ff1c56e9408b74bbbc0e6e3e5 WatchSource:0}: Error finding container 6d7b743479df15557bc9fd212a53eee26b2c3d3ff1c56e9408b74bbbc0e6e3e5: Status 404 returned error can't find the container with id 6d7b743479df15557bc9fd212a53eee26b2c3d3ff1c56e9408b74bbbc0e6e3e5
Jul 08 06:55:33.793703 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:33.793683    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-smjcg\""
Jul 08 06:55:33.801699 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:33.801679    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:55:33.913569 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:33.913542    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4bmr"]
Jul 08 06:55:33.916274 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:55:33.916248    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de691b9_d68b_4ae5_a0aa_47587f146d7e.slice/crio-44dbcf7b2565354db869bfc50fa7e66fdf503d855e2940c06fb4d618c017274c WatchSource:0}: Error finding container 44dbcf7b2565354db869bfc50fa7e66fdf503d855e2940c06fb4d618c017274c: Status 404 returned error can't find the container with id 44dbcf7b2565354db869bfc50fa7e66fdf503d855e2940c06fb4d618c017274c
Jul 08 06:55:34.018302 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:34.018267    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4bmr" event={"ID":"8de691b9-d68b-4ae5-a0aa-47587f146d7e","Type":"ContainerStarted","Data":"44dbcf7b2565354db869bfc50fa7e66fdf503d855e2940c06fb4d618c017274c"}
Jul 08 06:55:34.019403 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:34.019379    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7zf27" event={"ID":"39d02a2c-8802-46ce-a765-576cca29608e","Type":"ContainerStarted","Data":"6d7b743479df15557bc9fd212a53eee26b2c3d3ff1c56e9408b74bbbc0e6e3e5"}
Jul 08 06:55:34.934897 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:34.934859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:55:34.935368 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:34.935034    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:55:34.935368 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:55:34.935100    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls podName:031afabe-d529-4539-b152-d45a7c651d42 nodeName:}" failed. No retries permitted until 2026-07-08 06:56:06.935080966 +0000 UTC m=+99.671425681 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls") pod "dns-default-gtnj4" (UID: "031afabe-d529-4539-b152-d45a7c651d42") : secret "dns-default-metrics-tls" not found
Jul 08 06:55:39.034292 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:39.034256    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4bmr" event={"ID":"8de691b9-d68b-4ae5-a0aa-47587f146d7e","Type":"ContainerStarted","Data":"73152741b3d0fdaf97e65dc91710905e4c6bf1bf90a60c8713306a1bb5a6e55a"}
Jul 08 06:55:39.034733 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:39.034353    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:55:39.035817 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:39.035795    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7zf27" event={"ID":"39d02a2c-8802-46ce-a765-576cca29608e","Type":"ContainerStarted","Data":"17ec3e6583d129430f32bff0a0e56637378a01be1d32000aefb146adf0451117"}
Jul 08 06:55:39.052146 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:39.052101    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h4bmr" podStartSLOduration=67.397336204 podStartE2EDuration="1m12.052089196s" totalImagesPullingTime="4.654752992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:39.051459819 +0000 UTC m=+71.787804534" watchObservedRunningTime="2026-07-08 06:55:39.052089196 +0000 UTC m=+71.788433909"
Jul 08 06:55:39.065442 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:55:39.065404    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7zf27" podStartSLOduration=67.206612539 podStartE2EDuration="1m12.065391782s" totalImagesPullingTime="4.858779243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:55:39.064891964 +0000 UTC m=+71.801236693" watchObservedRunningTime="2026-07-08 06:55:39.065391782 +0000 UTC m=+71.801736495"
Jul 08 06:56:06.937910 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:56:06.937800    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:56:06.938388 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:56:06.937925    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:56:06.938388 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:56:06.937978    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls podName:031afabe-d529-4539-b152-d45a7c651d42 nodeName:}" failed. No retries permitted until 2026-07-08 06:57:10.937965089 +0000 UTC m=+163.674309782 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls") pod "dns-default-gtnj4" (UID: "031afabe-d529-4539-b152-d45a7c651d42") : secret "dns-default-metrics-tls" not found
Jul 08 06:56:10.040329 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:56:10.040302    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h4bmr"
Jul 08 06:56:37.622105 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:56:37.622071    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:56:37.622570 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:56:37.622202    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 06:56:37.622570 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:56:37.622262    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs podName:77090b45-e309-490a-973e-28c723351e73 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:39.62224725 +0000 UTC m=+252.358591942 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs") pod "network-metrics-daemon-r4hnb" (UID: "77090b45-e309-490a-973e-28c723351e73") : secret "metrics-daemon-secret" not found
Jul 08 06:57:06.069918 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:57:06.069879    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gtnj4" podUID="031afabe-d529-4539-b152-d45a7c651d42"
Jul 08 06:57:06.245000 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:57:06.244971    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:57:07.789570 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:57:07.789532    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:57:11.028474 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:57:11.028443    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:57:11.028879 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:57:11.028548    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 06:57:11.028879 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:57:11.028599    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls podName:031afabe-d529-4539-b152-d45a7c651d42 nodeName:}" failed. No retries permitted until 2026-07-08 06:59:13.028586297 +0000 UTC m=+285.764930990 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls") pod "dns-default-gtnj4" (UID: "031afabe-d529-4539-b152-d45a7c651d42") : secret "dns-default-metrics-tls" not found
Jul 08 06:57:21.780315 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:57:21.780231    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:58:39.683935 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:39.683885    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:58:39.684436 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:39.684033    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 08 06:58:39.684436 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:39.684102    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs podName:77090b45-e309-490a-973e-28c723351e73 nodeName:}" failed. No retries permitted until 2026-07-08 07:00:41.684086846 +0000 UTC m=+374.420431539 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs") pod "network-metrics-daemon-r4hnb" (UID: "77090b45-e309-490a-973e-28c723351e73") : secret "metrics-daemon-secret" not found
Jul 08 06:58:45.170593 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.170556    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv"]
Jul 08 06:58:45.172567 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.172552    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-5hz6v"]
Jul 08 06:58:45.172699 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.172682    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv"
Jul 08 06:58:45.174303 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.174283    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.175608 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.175589    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 08 06:58:45.175876 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.175862    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-dbhrl\""
Jul 08 06:58:45.176107 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.176092    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 08 06:58:45.176107 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.176101    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 08 06:58:45.176747 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.176732    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-d4f5z\""
Jul 08 06:58:45.178432 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.178417    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 08 06:58:45.178527 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.178436    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 08 06:58:45.178527 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.178455    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 08 06:58:45.179071 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.179052    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 08 06:58:45.179457 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.179256    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 08 06:58:45.188384 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.188367    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 08 06:58:45.191638 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.191619    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv"]
Jul 08 06:58:45.195328 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.195312    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-5hz6v"]
Jul 08 06:58:45.218831 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.218810    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzrg4\" (UniqueName: \"kubernetes.io/projected/1617fdda-eaa0-41c3-b37f-220f8f43c63e-kube-api-access-kzrg4\") pod \"kube-storage-version-migrator-operator-846bf668f7-wpkpv\" (UID: \"1617fdda-eaa0-41c3-b37f-220f8f43c63e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv"
Jul 08 06:58:45.218924 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.218836    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f78e9976-f78f-4aca-a985-f2e3d54dbc64-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.218924 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.218853    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f78e9976-f78f-4aca-a985-f2e3d54dbc64-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.218924 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.218883    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f78e9976-f78f-4aca-a985-f2e3d54dbc64-tmp\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.219089 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.218967    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1617fdda-eaa0-41c3-b37f-220f8f43c63e-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-wpkpv\" (UID: \"1617fdda-eaa0-41c3-b37f-220f8f43c63e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv"
Jul 08 06:58:45.219089 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.218997    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1617fdda-eaa0-41c3-b37f-220f8f43c63e-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-wpkpv\" (UID: \"1617fdda-eaa0-41c3-b37f-220f8f43c63e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv"
Jul 08 06:58:45.219089 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.219022    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqhk8\" (UniqueName: \"kubernetes.io/projected/f78e9976-f78f-4aca-a985-f2e3d54dbc64-kube-api-access-nqhk8\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.219089 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.219048    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f78e9976-f78f-4aca-a985-f2e3d54dbc64-snapshots\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.219089 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.219080    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f78e9976-f78f-4aca-a985-f2e3d54dbc64-serving-cert\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.226616 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.226599    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-mwb6f"]
Jul 08 06:58:45.228351 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.228338    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mwb6f"
Jul 08 06:58:45.229843 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.229825    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"]
Jul 08 06:58:45.230903 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.230888    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9xfg2\""
Jul 08 06:58:45.231666 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.231650    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"]
Jul 08 06:58:45.231787 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.231774    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:58:45.233489 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.233474    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"
Jul 08 06:58:45.237528 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.237506    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 08 06:58:45.238143 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.238118    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 08 06:58:45.238684 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.238663    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 08 06:58:45.239119 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.239097    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 08 06:58:45.239411 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.239133    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 08 06:58:45.239936 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.239914    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ncqmh\""
Jul 08 06:58:45.240032 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.240009    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-642mr\""
Jul 08 06:58:45.240166 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.240114    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 08 06:58:45.240242 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.240217    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 08 06:58:45.240583 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.240566    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74bb4d45ff-fzccm"]
Jul 08 06:58:45.242261 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.242245    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.245168 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.245148    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 08 06:58:45.245353 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.245336    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 08 06:58:45.245665 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.245645    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-mwb6f"]
Jul 08 06:58:45.245873 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.245855    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5p27c\""
Jul 08 06:58:45.246068 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.246052    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 08 06:58:45.249046 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.249022    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"]
Jul 08 06:58:45.250822 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.250568    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"]
Jul 08 06:58:45.251744 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.251723    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 08 06:58:45.257494 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.257468    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74bb4d45ff-fzccm"]
Jul 08 06:58:45.319982 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.319954    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f78e9976-f78f-4aca-a985-f2e3d54dbc64-serving-cert\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.320105 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320001    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-bound-sa-token\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.320105 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320028    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f78e9976-f78f-4aca-a985-f2e3d54dbc64-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.320105 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320060    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f78e9976-f78f-4aca-a985-f2e3d54dbc64-tmp\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.320105 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320086    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-certificates\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.320276 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320110    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-installation-pull-secrets\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.320276 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320190    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:58:45.320276 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320223    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lbv9\" (UniqueName: \"kubernetes.io/projected/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-kube-api-access-9lbv9\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:58:45.320276 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320254    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1617fdda-eaa0-41c3-b37f-220f8f43c63e-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-wpkpv\" (UID: \"1617fdda-eaa0-41c3-b37f-220f8f43c63e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv"
Jul 08 06:58:45.320434 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320318    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:58:45.320434 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320369    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-image-registry-private-configuration\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.320434 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320416    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-trusted-ca\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.320434 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320425    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f78e9976-f78f-4aca-a985-f2e3d54dbc64-tmp\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.320574 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320441    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfgxz\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-kube-api-access-qfgxz\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.320574 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320468    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.320574 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320500    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzrg4\" (UniqueName: \"kubernetes.io/projected/1617fdda-eaa0-41c3-b37f-220f8f43c63e-kube-api-access-kzrg4\") pod \"kube-storage-version-migrator-operator-846bf668f7-wpkpv\" (UID: \"1617fdda-eaa0-41c3-b37f-220f8f43c63e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv"
Jul 08 06:58:45.320574 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320526    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfjsr\" (UniqueName: \"kubernetes.io/projected/372d33b3-653f-42b0-a478-538a4b2f7340-kube-api-access-bfjsr\") pod \"network-check-source-989f8b8dc-mwb6f\" (UID: \"372d33b3-653f-42b0-a478-538a4b2f7340\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mwb6f"
Jul 08 06:58:45.320574 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320553    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f78e9976-f78f-4aca-a985-f2e3d54dbc64-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.320839 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320589    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-ca-trust-extracted\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.320839 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320618    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7lsvs\" (UID: \"74b284c7-3b36-48fc-9c30-077a20898d05\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"
Jul 08 06:58:45.320839 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320660    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1617fdda-eaa0-41c3-b37f-220f8f43c63e-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-wpkpv\" (UID: \"1617fdda-eaa0-41c3-b37f-220f8f43c63e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv"
Jul 08 06:58:45.320839 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320690    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f78e9976-f78f-4aca-a985-f2e3d54dbc64-snapshots\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.320839 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320723    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqhk8\" (UniqueName: \"kubernetes.io/projected/f78e9976-f78f-4aca-a985-f2e3d54dbc64-kube-api-access-nqhk8\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.320839 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320752    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4v2f\" (UniqueName: \"kubernetes.io/projected/74b284c7-3b36-48fc-9c30-077a20898d05-kube-api-access-g4v2f\") pod \"cluster-samples-operator-56fcf86bf-7lsvs\" (UID: \"74b284c7-3b36-48fc-9c30-077a20898d05\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"
Jul 08 06:58:45.321094 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.320840    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1617fdda-eaa0-41c3-b37f-220f8f43c63e-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-wpkpv\" (UID: \"1617fdda-eaa0-41c3-b37f-220f8f43c63e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv"
Jul 08 06:58:45.321094 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.321088    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f78e9976-f78f-4aca-a985-f2e3d54dbc64-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.321389 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.321354    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f78e9976-f78f-4aca-a985-f2e3d54dbc64-snapshots\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.321478 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.321443    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f78e9976-f78f-4aca-a985-f2e3d54dbc64-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.322443 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.322424    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f78e9976-f78f-4aca-a985-f2e3d54dbc64-serving-cert\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.322893 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.322878    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1617fdda-eaa0-41c3-b37f-220f8f43c63e-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-wpkpv\" (UID: \"1617fdda-eaa0-41c3-b37f-220f8f43c63e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv"
Jul 08 06:58:45.329653 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.329631    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzrg4\" (UniqueName: \"kubernetes.io/projected/1617fdda-eaa0-41c3-b37f-220f8f43c63e-kube-api-access-kzrg4\") pod \"kube-storage-version-migrator-operator-846bf668f7-wpkpv\" (UID: \"1617fdda-eaa0-41c3-b37f-220f8f43c63e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv"
Jul 08 06:58:45.331808 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.331789    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqhk8\" (UniqueName: \"kubernetes.io/projected/f78e9976-f78f-4aca-a985-f2e3d54dbc64-kube-api-access-nqhk8\") pod \"insights-operator-5c847dcfc4-5hz6v\" (UID: \"f78e9976-f78f-4aca-a985-f2e3d54dbc64\") " pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.421938 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.421876    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-ca-trust-extracted\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.421938 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.421910    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7lsvs\" (UID: \"74b284c7-3b36-48fc-9c30-077a20898d05\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"
Jul 08 06:58:45.422081 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.421950    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4v2f\" (UniqueName: \"kubernetes.io/projected/74b284c7-3b36-48fc-9c30-077a20898d05-kube-api-access-g4v2f\") pod \"cluster-samples-operator-56fcf86bf-7lsvs\" (UID: \"74b284c7-3b36-48fc-9c30-077a20898d05\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"
Jul 08 06:58:45.422081 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.421992    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-bound-sa-token\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.422081 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:45.422023    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 06:58:45.422235 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:45.422128    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls podName:74b284c7-3b36-48fc-9c30-077a20898d05 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:45.922111552 +0000 UTC m=+258.658456247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-7lsvs" (UID: "74b284c7-3b36-48fc-9c30-077a20898d05") : secret "samples-operator-tls" not found
Jul 08 06:58:45.422235 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.422026    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-certificates\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.422235 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.422179    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-installation-pull-secrets\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.422235 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.422213    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-ca-trust-extracted\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.422235 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.422226    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:58:45.422486 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.422251    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lbv9\" (UniqueName: \"kubernetes.io/projected/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-kube-api-access-9lbv9\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:58:45.422486 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.422272    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:58:45.422486 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.422288    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-image-registry-private-configuration\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.422486 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.422308    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-trusted-ca\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.422486 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.422324    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfgxz\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-kube-api-access-qfgxz\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.422486 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.422342    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.422486 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.422360    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfjsr\" (UniqueName: \"kubernetes.io/projected/372d33b3-653f-42b0-a478-538a4b2f7340-kube-api-access-bfjsr\") pod \"network-check-source-989f8b8dc-mwb6f\" (UID: \"372d33b3-653f-42b0-a478-538a4b2f7340\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mwb6f"
Jul 08 06:58:45.422486 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:45.422447    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 06:58:45.422845 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:45.422507    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls podName:a95c5ec7-df5b-4792-b40d-f43d9cf90ba8 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:45.922491074 +0000 UTC m=+258.658835766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-dpkd6" (UID: "a95c5ec7-df5b-4792-b40d-f43d9cf90ba8") : secret "cluster-monitoring-operator-tls" not found
Jul 08 06:58:45.422845 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:45.422526    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 06:58:45.422845 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:45.422543    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74bb4d45ff-fzccm: secret "image-registry-tls" not found
Jul 08 06:58:45.422845 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:45.422597    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls podName:ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:45.922580739 +0000 UTC m=+258.658925432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls") pod "image-registry-74bb4d45ff-fzccm" (UID: "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67") : secret "image-registry-tls" not found
Jul 08 06:58:45.422845 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.422665    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-certificates\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.423307 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.422973    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:58:45.423307 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.423129    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-trusted-ca\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.424283 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.424264    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-installation-pull-secrets\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.424743 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.424727    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-image-registry-private-configuration\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.432953 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.432926    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfgxz\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-kube-api-access-qfgxz\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.433691 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.433672    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-bound-sa-token\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.433807 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.433792    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4v2f\" (UniqueName: \"kubernetes.io/projected/74b284c7-3b36-48fc-9c30-077a20898d05-kube-api-access-g4v2f\") pod \"cluster-samples-operator-56fcf86bf-7lsvs\" (UID: \"74b284c7-3b36-48fc-9c30-077a20898d05\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"
Jul 08 06:58:45.434048 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.434034    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfjsr\" (UniqueName: \"kubernetes.io/projected/372d33b3-653f-42b0-a478-538a4b2f7340-kube-api-access-bfjsr\") pod \"network-check-source-989f8b8dc-mwb6f\" (UID: \"372d33b3-653f-42b0-a478-538a4b2f7340\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mwb6f"
Jul 08 06:58:45.434229 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.434215    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lbv9\" (UniqueName: \"kubernetes.io/projected/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-kube-api-access-9lbv9\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:58:45.482051 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.482032    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv"
Jul 08 06:58:45.487050 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.487035    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v"
Jul 08 06:58:45.539585 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.539428    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mwb6f"
Jul 08 06:58:45.616592 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.616565    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-5hz6v"]
Jul 08 06:58:45.619979 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:58:45.619953    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78e9976_f78f_4aca_a985_f2e3d54dbc64.slice/crio-84b620cd7c9eba2a06a662afcdb6638909d60f0f4d0f7555a961a61ea998b9dc WatchSource:0}: Error finding container 84b620cd7c9eba2a06a662afcdb6638909d60f0f4d0f7555a961a61ea998b9dc: Status 404 returned error can't find the container with id 84b620cd7c9eba2a06a662afcdb6638909d60f0f4d0f7555a961a61ea998b9dc
Jul 08 06:58:45.628917 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.628491    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv"]
Jul 08 06:58:45.632413 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:58:45.632388    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1617fdda_eaa0_41c3_b37f_220f8f43c63e.slice/crio-81de4b4dc5bc56bbd48c58115d5b85ea1799cb90aafa5532830d6c8e797a9f09 WatchSource:0}: Error finding container 81de4b4dc5bc56bbd48c58115d5b85ea1799cb90aafa5532830d6c8e797a9f09: Status 404 returned error can't find the container with id 81de4b4dc5bc56bbd48c58115d5b85ea1799cb90aafa5532830d6c8e797a9f09
Jul 08 06:58:45.673958 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.673939    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-mwb6f"]
Jul 08 06:58:45.675966 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:58:45.675942    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372d33b3_653f_42b0_a478_538a4b2f7340.slice/crio-0283edb37f86377512613954c1992ce890af3b963be60b50a5a473c9401472b6 WatchSource:0}: Error finding container 0283edb37f86377512613954c1992ce890af3b963be60b50a5a473c9401472b6: Status 404 returned error can't find the container with id 0283edb37f86377512613954c1992ce890af3b963be60b50a5a473c9401472b6
Jul 08 06:58:45.927184 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.927108    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7lsvs\" (UID: \"74b284c7-3b36-48fc-9c30-077a20898d05\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"
Jul 08 06:58:45.927184 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.927176    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:58:45.927397 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:45.927205    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:45.927397 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:45.927283    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 06:58:45.927397 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:45.927323    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 06:58:45.927397 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:45.927335    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74bb4d45ff-fzccm: secret "image-registry-tls" not found
Jul 08 06:58:45.927397 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:45.927334    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 06:58:45.927397 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:45.927358    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls podName:74b284c7-3b36-48fc-9c30-077a20898d05 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:46.927336508 +0000 UTC m=+259.663681204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-7lsvs" (UID: "74b284c7-3b36-48fc-9c30-077a20898d05") : secret "samples-operator-tls" not found
Jul 08 06:58:45.927397 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:45.927376    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls podName:ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:46.92736756 +0000 UTC m=+259.663712253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls") pod "image-registry-74bb4d45ff-fzccm" (UID: "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67") : secret "image-registry-tls" not found
Jul 08 06:58:45.927397 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:45.927393    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls podName:a95c5ec7-df5b-4792-b40d-f43d9cf90ba8 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:46.927382944 +0000 UTC m=+259.663727641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-dpkd6" (UID: "a95c5ec7-df5b-4792-b40d-f43d9cf90ba8") : secret "cluster-monitoring-operator-tls" not found
Jul 08 06:58:46.481914 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:46.481775    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mwb6f" event={"ID":"372d33b3-653f-42b0-a478-538a4b2f7340","Type":"ContainerStarted","Data":"19143a12c9735d111da6a14a3dd845000b108dc75f0f9d1683e4c0d3241c1be6"}
Jul 08 06:58:46.481914 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:46.481845    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mwb6f" event={"ID":"372d33b3-653f-42b0-a478-538a4b2f7340","Type":"ContainerStarted","Data":"0283edb37f86377512613954c1992ce890af3b963be60b50a5a473c9401472b6"}
Jul 08 06:58:46.483467 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:46.483435    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v" event={"ID":"f78e9976-f78f-4aca-a985-f2e3d54dbc64","Type":"ContainerStarted","Data":"84b620cd7c9eba2a06a662afcdb6638909d60f0f4d0f7555a961a61ea998b9dc"}
Jul 08 06:58:46.484750 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:46.484710    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv" event={"ID":"1617fdda-eaa0-41c3-b37f-220f8f43c63e","Type":"ContainerStarted","Data":"81de4b4dc5bc56bbd48c58115d5b85ea1799cb90aafa5532830d6c8e797a9f09"}
Jul 08 06:58:46.497787 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:46.497715    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-mwb6f" podStartSLOduration=1.497697104 podStartE2EDuration="1.497697104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:58:46.497211598 +0000 UTC m=+259.233556312" watchObservedRunningTime="2026-07-08 06:58:46.497697104 +0000 UTC m=+259.234041819"
Jul 08 06:58:46.938096 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:46.938013    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:58:46.938096 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:46.938080    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:46.938313 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:46.938124    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7lsvs\" (UID: \"74b284c7-3b36-48fc-9c30-077a20898d05\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"
Jul 08 06:58:46.938313 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:46.938149    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 06:58:46.941792 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:46.938582    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 06:58:46.941792 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:46.938685    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls podName:74b284c7-3b36-48fc-9c30-077a20898d05 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:48.938663073 +0000 UTC m=+261.675007774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-7lsvs" (UID: "74b284c7-3b36-48fc-9c30-077a20898d05") : secret "samples-operator-tls" not found
Jul 08 06:58:46.941792 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:46.938733    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls podName:a95c5ec7-df5b-4792-b40d-f43d9cf90ba8 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:48.938707515 +0000 UTC m=+261.675052220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-dpkd6" (UID: "a95c5ec7-df5b-4792-b40d-f43d9cf90ba8") : secret "cluster-monitoring-operator-tls" not found
Jul 08 06:58:46.941792 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:46.938739    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 06:58:46.941792 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:46.938778    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74bb4d45ff-fzccm: secret "image-registry-tls" not found
Jul 08 06:58:46.941792 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:46.938833    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls podName:ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:48.938819101 +0000 UTC m=+261.675163795 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls") pod "image-registry-74bb4d45ff-fzccm" (UID: "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67") : secret "image-registry-tls" not found
Jul 08 06:58:48.492627 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:48.492592    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v" event={"ID":"f78e9976-f78f-4aca-a985-f2e3d54dbc64","Type":"ContainerStarted","Data":"4c6e28c52de0d510fe5eb57a7c1c749f2f26f4bfb1a0142d94a2d6c3f8ca963b"}
Jul 08 06:58:48.494015 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:48.493993    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv" event={"ID":"1617fdda-eaa0-41c3-b37f-220f8f43c63e","Type":"ContainerStarted","Data":"8589b7f6dfb49cbf12f54fa69be3bc9466abdd2b6e50630094cabdb4b10fbd80"}
Jul 08 06:58:48.510082 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:48.510043    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v" podStartSLOduration=1.02772917 podStartE2EDuration="3.51003114s" totalImagesPullingTime="2.48230197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:58:48.508427678 +0000 UTC m=+261.244772393" watchObservedRunningTime="2026-07-08 06:58:48.51003114 +0000 UTC m=+261.246375831"
Jul 08 06:58:48.525951 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:48.525911    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv" podStartSLOduration=1.056820801 podStartE2EDuration="3.525898759s" totalImagesPullingTime="2.469077958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:58:48.524914175 +0000 UTC m=+261.261258893" watchObservedRunningTime="2026-07-08 06:58:48.525898759 +0000 UTC m=+261.262243473"
Jul 08 06:58:48.955048 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:48.954966    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7lsvs\" (UID: \"74b284c7-3b36-48fc-9c30-077a20898d05\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"
Jul 08 06:58:48.955209 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:48.955063    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:58:48.955209 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:48.955095    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:48.955209 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:48.955141    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 06:58:48.955209 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:48.955188    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 06:58:48.955209 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:48.955194    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 06:58:48.955209 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:48.955210    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74bb4d45ff-fzccm: secret "image-registry-tls" not found
Jul 08 06:58:48.955418 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:48.955216    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls podName:74b284c7-3b36-48fc-9c30-077a20898d05 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:52.95519734 +0000 UTC m=+265.691542038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-7lsvs" (UID: "74b284c7-3b36-48fc-9c30-077a20898d05") : secret "samples-operator-tls" not found
Jul 08 06:58:48.955418 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:48.955237    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls podName:a95c5ec7-df5b-4792-b40d-f43d9cf90ba8 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:52.955224785 +0000 UTC m=+265.691569476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-dpkd6" (UID: "a95c5ec7-df5b-4792-b40d-f43d9cf90ba8") : secret "cluster-monitoring-operator-tls" not found
Jul 08 06:58:48.955418 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:48.955251    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls podName:ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:52.955244686 +0000 UTC m=+265.691589377 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls") pod "image-registry-74bb4d45ff-fzccm" (UID: "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67") : secret "image-registry-tls" not found
Jul 08 06:58:49.223823 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.223794    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vbzqz"]
Jul 08 06:58:49.225953 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.225938    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.230893 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.230867    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 08 06:58:49.231017 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.230899    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 08 06:58:49.231017 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.230903    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qvkm5\""
Jul 08 06:58:49.243952 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.243927    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vbzqz"]
Jul 08 06:58:49.358728 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.358699    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcq8w\" (UniqueName: \"kubernetes.io/projected/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-api-access-kcq8w\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.358861 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.358735    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.358861 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.358783    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.358861 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.358827    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4e676c1-8726-4666-b3b7-23eb52cb32a1-crio-socket\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.358861 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.358852    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4e676c1-8726-4666-b3b7-23eb52cb32a1-data-volume\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.459950 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.459924    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcq8w\" (UniqueName: \"kubernetes.io/projected/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-api-access-kcq8w\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.459950 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.459952    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.460109 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.459970    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.460109 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.459989    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4e676c1-8726-4666-b3b7-23eb52cb32a1-crio-socket\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.460109 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.460043    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c4e676c1-8726-4666-b3b7-23eb52cb32a1-crio-socket\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.460109 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:49.460055    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 06:58:49.460109 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:49.460098    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 06:58:49.460312 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.460110    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4e676c1-8726-4666-b3b7-23eb52cb32a1-data-volume\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.460312 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:49.460122    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm podName:c4e676c1-8726-4666-b3b7-23eb52cb32a1 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:49.960104404 +0000 UTC m=+262.696449102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vbzqz" (UID: "c4e676c1-8726-4666-b3b7-23eb52cb32a1") : configmap "kube-rbac-proxy" not found
Jul 08 06:58:49.460312 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:49.460154    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls podName:c4e676c1-8726-4666-b3b7-23eb52cb32a1 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:49.960136979 +0000 UTC m=+262.696481671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vbzqz" (UID: "c4e676c1-8726-4666-b3b7-23eb52cb32a1") : secret "insights-runtime-extractor-tls" not found
Jul 08 06:58:49.460418 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.460385    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c4e676c1-8726-4666-b3b7-23eb52cb32a1-data-volume\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.470119 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.470096    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcq8w\" (UniqueName: \"kubernetes.io/projected/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-api-access-kcq8w\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.964512 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.964480    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.964512 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:49.964511    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:49.964912 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:49.964598    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 06:58:49.964912 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:49.964614    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 06:58:49.964912 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:49.964659    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm podName:c4e676c1-8726-4666-b3b7-23eb52cb32a1 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:50.964643324 +0000 UTC m=+263.700988016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vbzqz" (UID: "c4e676c1-8726-4666-b3b7-23eb52cb32a1") : configmap "kube-rbac-proxy" not found
Jul 08 06:58:49.964912 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:49.964675    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls podName:c4e676c1-8726-4666-b3b7-23eb52cb32a1 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:50.964667153 +0000 UTC m=+263.701011845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vbzqz" (UID: "c4e676c1-8726-4666-b3b7-23eb52cb32a1") : secret "insights-runtime-extractor-tls" not found
Jul 08 06:58:50.857348 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:50.857313    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-7d5tw"]
Jul 08 06:58:50.859342 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:50.859327    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7d5tw"
Jul 08 06:58:50.861735 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:50.861714    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vsgbw\""
Jul 08 06:58:50.861735 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:50.861719    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 08 06:58:50.862716 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:50.862691    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 08 06:58:50.867748 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:50.867729    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-7d5tw"]
Jul 08 06:58:50.972571 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:50.972547    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:50.972961 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:50.972577    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:50.972961 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:50.972620    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf92v\" (UniqueName: \"kubernetes.io/projected/fd6db342-303a-4506-bcc3-5916d051cc03-kube-api-access-tf92v\") pod \"migrator-d57d58cc9-7d5tw\" (UID: \"fd6db342-303a-4506-bcc3-5916d051cc03\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7d5tw"
Jul 08 06:58:50.972961 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:50.972680    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 06:58:50.972961 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:50.972708    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 06:58:50.972961 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:50.972777    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm podName:c4e676c1-8726-4666-b3b7-23eb52cb32a1 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:52.972733646 +0000 UTC m=+265.709078358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vbzqz" (UID: "c4e676c1-8726-4666-b3b7-23eb52cb32a1") : configmap "kube-rbac-proxy" not found
Jul 08 06:58:50.972961 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:50.972801    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls podName:c4e676c1-8726-4666-b3b7-23eb52cb32a1 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:52.972790744 +0000 UTC m=+265.709135441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vbzqz" (UID: "c4e676c1-8726-4666-b3b7-23eb52cb32a1") : secret "insights-runtime-extractor-tls" not found
Jul 08 06:58:50.982732 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:50.982711    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6gzn7_a4f29e73-17e3-42cd-8bb5-980dfe3729c2/dns-node-resolver/0.log"
Jul 08 06:58:51.073957 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:51.073931    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf92v\" (UniqueName: \"kubernetes.io/projected/fd6db342-303a-4506-bcc3-5916d051cc03-kube-api-access-tf92v\") pod \"migrator-d57d58cc9-7d5tw\" (UID: \"fd6db342-303a-4506-bcc3-5916d051cc03\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7d5tw"
Jul 08 06:58:51.083466 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:51.083443    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf92v\" (UniqueName: \"kubernetes.io/projected/fd6db342-303a-4506-bcc3-5916d051cc03-kube-api-access-tf92v\") pod \"migrator-d57d58cc9-7d5tw\" (UID: \"fd6db342-303a-4506-bcc3-5916d051cc03\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7d5tw"
Jul 08 06:58:51.169488 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:51.169421    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7d5tw"
Jul 08 06:58:51.290738 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:51.290713    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-7d5tw"]
Jul 08 06:58:51.293577 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:58:51.293541    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6db342_303a_4506_bcc3_5916d051cc03.slice/crio-97b8e75ca42cfa957700898d0cdb733b2d1e9e4e22a763b711f448b52269c1cb WatchSource:0}: Error finding container 97b8e75ca42cfa957700898d0cdb733b2d1e9e4e22a763b711f448b52269c1cb: Status 404 returned error can't find the container with id 97b8e75ca42cfa957700898d0cdb733b2d1e9e4e22a763b711f448b52269c1cb
Jul 08 06:58:51.503836 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:51.503803    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7d5tw" event={"ID":"fd6db342-303a-4506-bcc3-5916d051cc03","Type":"ContainerStarted","Data":"97b8e75ca42cfa957700898d0cdb733b2d1e9e4e22a763b711f448b52269c1cb"}
Jul 08 06:58:52.181344 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:52.181319    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d5qzh_fd567957-630a-4c89-8890-7abcf95e7298/node-ca/0.log"
Jul 08 06:58:52.510106 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:52.510069    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7d5tw" event={"ID":"fd6db342-303a-4506-bcc3-5916d051cc03","Type":"ContainerStarted","Data":"a9e30b6df3193e85473a6bf7f15bc46aad060f804e551d36ac1eb5ca4b16d74b"}
Jul 08 06:58:52.510226 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:52.510130    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7d5tw" event={"ID":"fd6db342-303a-4506-bcc3-5916d051cc03","Type":"ContainerStarted","Data":"63ce7a4a9a9d570791ef652b710742a75840caf87ee0584b969466495f2681c3"}
Jul 08 06:58:52.523761 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:52.523722    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-7d5tw" podStartSLOduration=1.491773829 podStartE2EDuration="2.523707971s" totalImagesPullingTime="1.031934142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:58:52.523433763 +0000 UTC m=+265.259778477" watchObservedRunningTime="2026-07-08 06:58:52.523707971 +0000 UTC m=+265.260052685"
Jul 08 06:58:52.981115 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:52.981088    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lcn7q_4847909b-7f1a-44f2-9f42-d017286c2d6d/serve-healthcheck-canary/0.log"
Jul 08 06:58:52.989247 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:52.989218    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:52.989371 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:52.989256    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:52.989371 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:52.989290    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:58:52.989371 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:52.989310    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 06:58:52.989371 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:52.989322    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:58:52.989371 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:52.989364    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm podName:c4e676c1-8726-4666-b3b7-23eb52cb32a1 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:56.989346826 +0000 UTC m=+269.725691518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vbzqz" (UID: "c4e676c1-8726-4666-b3b7-23eb52cb32a1") : configmap "kube-rbac-proxy" not found
Jul 08 06:58:52.989608 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:52.989386    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7lsvs\" (UID: \"74b284c7-3b36-48fc-9c30-077a20898d05\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"
Jul 08 06:58:52.989608 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:52.989428    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 06:58:52.989608 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:52.989440    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 06:58:52.989608 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:52.989456    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74bb4d45ff-fzccm: secret "image-registry-tls" not found
Jul 08 06:58:52.989608 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:52.989431    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 06:58:52.989608 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:52.989481    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls podName:c4e676c1-8726-4666-b3b7-23eb52cb32a1 nodeName:}" failed. No retries permitted until 2026-07-08 06:58:56.989465905 +0000 UTC m=+269.725810601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vbzqz" (UID: "c4e676c1-8726-4666-b3b7-23eb52cb32a1") : secret "insights-runtime-extractor-tls" not found
Jul 08 06:58:52.989608 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:52.989493    2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 08 06:58:52.989608 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:52.989505    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls podName:ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67 nodeName:}" failed. No retries permitted until 2026-07-08 06:59:00.989488294 +0000 UTC m=+273.725833008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls") pod "image-registry-74bb4d45ff-fzccm" (UID: "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67") : secret "image-registry-tls" not found
Jul 08 06:58:52.989608 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:52.989521    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls podName:a95c5ec7-df5b-4792-b40d-f43d9cf90ba8 nodeName:}" failed. No retries permitted until 2026-07-08 06:59:00.98951215 +0000 UTC m=+273.725856847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-dpkd6" (UID: "a95c5ec7-df5b-4792-b40d-f43d9cf90ba8") : secret "cluster-monitoring-operator-tls" not found
Jul 08 06:58:52.989608 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:52.989535    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls podName:74b284c7-3b36-48fc-9c30-077a20898d05 nodeName:}" failed. No retries permitted until 2026-07-08 06:59:00.989527191 +0000 UTC m=+273.725871883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-7lsvs" (UID: "74b284c7-3b36-48fc-9c30-077a20898d05") : secret "samples-operator-tls" not found
Jul 08 06:58:57.019552 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:57.019508    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:57.019552 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:58:57.019552    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:58:57.020018 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:57.019639    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 06:58:57.020018 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:57.019673    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 06:58:57.020018 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:57.019730    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm podName:c4e676c1-8726-4666-b3b7-23eb52cb32a1 nodeName:}" failed. No retries permitted until 2026-07-08 06:59:05.01970781 +0000 UTC m=+277.756052504 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vbzqz" (UID: "c4e676c1-8726-4666-b3b7-23eb52cb32a1") : configmap "kube-rbac-proxy" not found
Jul 08 06:58:57.020018 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:58:57.019751    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls podName:c4e676c1-8726-4666-b3b7-23eb52cb32a1 nodeName:}" failed. No retries permitted until 2026-07-08 06:59:05.019741324 +0000 UTC m=+277.756086015 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vbzqz" (UID: "c4e676c1-8726-4666-b3b7-23eb52cb32a1") : secret "insights-runtime-extractor-tls" not found
Jul 08 06:59:01.054778 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:01.054718    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:59:01.054778 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:01.054790    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:59:01.055293 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:01.054821    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7lsvs\" (UID: \"74b284c7-3b36-48fc-9c30-077a20898d05\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"
Jul 08 06:59:01.055293 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:01.054890    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 08 06:59:01.055293 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:01.054987    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls podName:a95c5ec7-df5b-4792-b40d-f43d9cf90ba8 nodeName:}" failed. No retries permitted until 2026-07-08 06:59:17.054966318 +0000 UTC m=+289.791311024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-dpkd6" (UID: "a95c5ec7-df5b-4792-b40d-f43d9cf90ba8") : secret "cluster-monitoring-operator-tls" not found
Jul 08 06:59:01.057259 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:01.057240    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/74b284c7-3b36-48fc-9c30-077a20898d05-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-7lsvs\" (UID: \"74b284c7-3b36-48fc-9c30-077a20898d05\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"
Jul 08 06:59:01.057367 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:01.057350    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls\") pod \"image-registry-74bb4d45ff-fzccm\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") " pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:59:01.154410 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:01.154369    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"
Jul 08 06:59:01.159372 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:01.159352    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:59:01.284408 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:01.284254    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs"]
Jul 08 06:59:01.302801 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:01.302777    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74bb4d45ff-fzccm"]
Jul 08 06:59:01.304921 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:59:01.304864    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6d20c7_5880_4d8e_9c18_7ed5ad9b9e67.slice/crio-147f1a7c73bbcb38baaa54f38c6d9b2270acf3e5c663ae81124d801359e96bf5 WatchSource:0}: Error finding container 147f1a7c73bbcb38baaa54f38c6d9b2270acf3e5c663ae81124d801359e96bf5: Status 404 returned error can't find the container with id 147f1a7c73bbcb38baaa54f38c6d9b2270acf3e5c663ae81124d801359e96bf5
Jul 08 06:59:01.538984 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:01.538946    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm" event={"ID":"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67","Type":"ContainerStarted","Data":"15fd5e3d99284299c52ba9a9b1d1ad4858f990d1ca411624a1c9dfca0f06a2cb"}
Jul 08 06:59:01.539146 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:01.539003    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm" event={"ID":"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67","Type":"ContainerStarted","Data":"147f1a7c73bbcb38baaa54f38c6d9b2270acf3e5c663ae81124d801359e96bf5"}
Jul 08 06:59:01.539225 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:01.539132    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:59:01.540096 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:01.540075    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs" event={"ID":"74b284c7-3b36-48fc-9c30-077a20898d05","Type":"ContainerStarted","Data":"282d0099b29f790ed42308785f8b10af64d943b4cac2e0a646b787f173750ffa"}
Jul 08 06:59:01.557367 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:01.557283    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm" podStartSLOduration=16.557270624 podStartE2EDuration="16.557270624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:59:01.556390306 +0000 UTC m=+274.292735019" watchObservedRunningTime="2026-07-08 06:59:01.557270624 +0000 UTC m=+274.293615337"
Jul 08 06:59:03.548320 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:03.548285    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs" event={"ID":"74b284c7-3b36-48fc-9c30-077a20898d05","Type":"ContainerStarted","Data":"8046ee3099f63b01d9689dc8373f1089f2bbd933a5de378312b75dcc5b4a7fb7"}
Jul 08 06:59:03.548686 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:03.548350    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs" event={"ID":"74b284c7-3b36-48fc-9c30-077a20898d05","Type":"ContainerStarted","Data":"5dd5960c71028d8cc19d564451584414c112e38d1464dcd7c8d3c6d94a995318"}
Jul 08 06:59:03.565490 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:03.565443    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-7lsvs" podStartSLOduration=17.098509921 podStartE2EDuration="18.565429591s" totalImagesPullingTime="1.46691967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:59:03.56414142 +0000 UTC m=+276.300486134" watchObservedRunningTime="2026-07-08 06:59:03.565429591 +0000 UTC m=+276.301774305"
Jul 08 06:59:05.088844 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:05.088803    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:59:05.088844 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:05.088841    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:59:05.089237 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:05.088967    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 06:59:05.089237 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:05.089017    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls podName:c4e676c1-8726-4666-b3b7-23eb52cb32a1 nodeName:}" failed. No retries permitted until 2026-07-08 06:59:21.08900411 +0000 UTC m=+293.825348802 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vbzqz" (UID: "c4e676c1-8726-4666-b3b7-23eb52cb32a1") : secret "insights-runtime-extractor-tls" not found
Jul 08 06:59:05.089313 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:05.089297    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c4e676c1-8726-4666-b3b7-23eb52cb32a1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:59:09.246399 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:09.246361    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gtnj4" podUID="031afabe-d529-4539-b152-d45a7c651d42"
Jul 08 06:59:09.568153 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:09.568077    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:59:13.046708 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:13.046660    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:59:13.049229 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:13.049207    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/031afabe-d529-4539-b152-d45a7c651d42-metrics-tls\") pod \"dns-default-gtnj4\" (UID: \"031afabe-d529-4539-b152-d45a7c651d42\") " pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:59:13.171232 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:13.171204    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vxwlb\""
Jul 08 06:59:13.179181 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:13.179162    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:59:13.297414 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:13.297348    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gtnj4"]
Jul 08 06:59:13.300413 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:59:13.300387    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031afabe_d529_4539_b152_d45a7c651d42.slice/crio-d48f714697beb4e3f3de1b29d25837b6998feb06438057de59e8f8cef9bcd6cd WatchSource:0}: Error finding container d48f714697beb4e3f3de1b29d25837b6998feb06438057de59e8f8cef9bcd6cd: Status 404 returned error can't find the container with id d48f714697beb4e3f3de1b29d25837b6998feb06438057de59e8f8cef9bcd6cd
Jul 08 06:59:13.581422 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:13.581327    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gtnj4" event={"ID":"031afabe-d529-4539-b152-d45a7c651d42","Type":"ContainerStarted","Data":"d48f714697beb4e3f3de1b29d25837b6998feb06438057de59e8f8cef9bcd6cd"}
Jul 08 06:59:15.438053 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:15.438021    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74bb4d45ff-fzccm"]
Jul 08 06:59:15.589989 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:15.589946    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gtnj4" event={"ID":"031afabe-d529-4539-b152-d45a7c651d42","Type":"ContainerStarted","Data":"d87569018bd88b281b946d53d43e3940515a013f46059fb299b89a0564f6aa5a"}
Jul 08 06:59:15.590145 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:15.590002    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gtnj4" event={"ID":"031afabe-d529-4539-b152-d45a7c651d42","Type":"ContainerStarted","Data":"703c2868e2d8a222633dea3e166fc95a670a28ee022d840061cb3ba53ee0a305"}
Jul 08 06:59:15.590145 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:15.590085    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:59:15.608562 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:15.608519    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gtnj4" podStartSLOduration=251.140857 podStartE2EDuration="4m12.608507513s" totalImagesPullingTime="1.467650513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:59:15.607667764 +0000 UTC m=+288.344012490" watchObservedRunningTime="2026-07-08 06:59:15.608507513 +0000 UTC m=+288.344852226"
Jul 08 06:59:17.077484 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:17.077449    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:59:17.079969 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:17.079946    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a95c5ec7-df5b-4792-b40d-f43d9cf90ba8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-dpkd6\" (UID: \"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:59:17.346595 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:17.346504    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"
Jul 08 06:59:17.469796 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:17.469746    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6"]
Jul 08 06:59:17.472362 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:59:17.472328    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95c5ec7_df5b_4792_b40d_f43d9cf90ba8.slice/crio-4492c86cb3d2b7c423692f80ef89a98f3a2f8184966a00bfbaf46b98284efbbd WatchSource:0}: Error finding container 4492c86cb3d2b7c423692f80ef89a98f3a2f8184966a00bfbaf46b98284efbbd: Status 404 returned error can't find the container with id 4492c86cb3d2b7c423692f80ef89a98f3a2f8184966a00bfbaf46b98284efbbd
Jul 08 06:59:17.599803 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:17.599713    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6" event={"ID":"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8","Type":"ContainerStarted","Data":"4492c86cb3d2b7c423692f80ef89a98f3a2f8184966a00bfbaf46b98284efbbd"}
Jul 08 06:59:19.529042 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:19.529014    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-jvth8"]
Jul 08 06:59:19.531970 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:19.531952    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-jvth8"
Jul 08 06:59:19.534366 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:19.534347    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7w8x5\""
Jul 08 06:59:19.534464 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:19.534345    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 08 06:59:19.541209 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:19.541190    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-jvth8"]
Jul 08 06:59:19.608232 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:19.608200    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6" event={"ID":"a95c5ec7-df5b-4792-b40d-f43d9cf90ba8","Type":"ContainerStarted","Data":"8b5dbb87b0d5c1e7bb35d38c8a73e625acf6ced9f46e9504a5db5e1a5a644a36"}
Jul 08 06:59:19.623616 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:19.623573    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-dpkd6" podStartSLOduration=33.031771892 podStartE2EDuration="34.623557053s" totalImagesPullingTime="1.591785161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:59:19.623382171 +0000 UTC m=+292.359726886" watchObservedRunningTime="2026-07-08 06:59:19.623557053 +0000 UTC m=+292.359901766"
Jul 08 06:59:19.697866 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:19.697830    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c115bfa0-b765-4f8b-a141-95b37f217b7c-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-jvth8\" (UID: \"c115bfa0-b765-4f8b-a141-95b37f217b7c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-jvth8"
Jul 08 06:59:19.799040 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:19.798977    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c115bfa0-b765-4f8b-a141-95b37f217b7c-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-jvth8\" (UID: \"c115bfa0-b765-4f8b-a141-95b37f217b7c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-jvth8"
Jul 08 06:59:19.799165 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:19.799091    2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 08 06:59:19.799165 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:19.799139    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c115bfa0-b765-4f8b-a141-95b37f217b7c-tls-certificates podName:c115bfa0-b765-4f8b-a141-95b37f217b7c nodeName:}" failed. No retries permitted until 2026-07-08 06:59:20.299126334 +0000 UTC m=+293.035471026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/c115bfa0-b765-4f8b-a141-95b37f217b7c-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-jvth8" (UID: "c115bfa0-b765-4f8b-a141-95b37f217b7c") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 08 06:59:20.302847 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:20.302813    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c115bfa0-b765-4f8b-a141-95b37f217b7c-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-jvth8\" (UID: \"c115bfa0-b765-4f8b-a141-95b37f217b7c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-jvth8"
Jul 08 06:59:20.307115 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:20.306246    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c115bfa0-b765-4f8b-a141-95b37f217b7c-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-jvth8\" (UID: \"c115bfa0-b765-4f8b-a141-95b37f217b7c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-jvth8"
Jul 08 06:59:20.441249 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:20.441219    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-jvth8"
Jul 08 06:59:20.561363 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:20.561276    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-jvth8"]
Jul 08 06:59:20.563829 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:59:20.563803    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc115bfa0_b765_4f8b_a141_95b37f217b7c.slice/crio-495f4468a0bb35f89ea534c69fcc583f67144ea127902092f7b2678012aa1b38 WatchSource:0}: Error finding container 495f4468a0bb35f89ea534c69fcc583f67144ea127902092f7b2678012aa1b38: Status 404 returned error can't find the container with id 495f4468a0bb35f89ea534c69fcc583f67144ea127902092f7b2678012aa1b38
Jul 08 06:59:20.612351 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:20.612320    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-jvth8" event={"ID":"c115bfa0-b765-4f8b-a141-95b37f217b7c","Type":"ContainerStarted","Data":"495f4468a0bb35f89ea534c69fcc583f67144ea127902092f7b2678012aa1b38"}
Jul 08 06:59:21.108808 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:21.108752    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:59:21.111364 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:21.111340    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c4e676c1-8726-4666-b3b7-23eb52cb32a1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vbzqz\" (UID: \"c4e676c1-8726-4666-b3b7-23eb52cb32a1\") " pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:59:21.335475 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:21.335443    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vbzqz"
Jul 08 06:59:21.671966 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:21.671943    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vbzqz"]
Jul 08 06:59:21.674350 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:59:21.674322    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e676c1_8726_4666_b3b7_23eb52cb32a1.slice/crio-5ddabb5b356fae32129a7d641958d830acda763519fe94786db5425481fb95ad WatchSource:0}: Error finding container 5ddabb5b356fae32129a7d641958d830acda763519fe94786db5425481fb95ad: Status 404 returned error can't find the container with id 5ddabb5b356fae32129a7d641958d830acda763519fe94786db5425481fb95ad
Jul 08 06:59:22.623458 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:22.623421    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vbzqz" event={"ID":"c4e676c1-8726-4666-b3b7-23eb52cb32a1","Type":"ContainerStarted","Data":"5ddabb5b356fae32129a7d641958d830acda763519fe94786db5425481fb95ad"}
Jul 08 06:59:22.625383 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:22.625352    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-jvth8" event={"ID":"c115bfa0-b765-4f8b-a141-95b37f217b7c","Type":"ContainerStarted","Data":"9f6b21d3c665f2944292e3c04c61ad2c9534bb5e4a1febd466d8ddc5ea5663aa"}
Jul 08 06:59:22.625913 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:22.625865    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-jvth8"
Jul 08 06:59:22.632827 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:22.632805    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-jvth8"
Jul 08 06:59:22.642417 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:22.642370    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-jvth8" podStartSLOduration=2.6064770299999998 podStartE2EDuration="3.642355666s" totalImagesPullingTime="1.035878636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:59:22.641362951 +0000 UTC m=+295.377707668" watchObservedRunningTime="2026-07-08 06:59:22.642355666 +0000 UTC m=+295.378700380"
Jul 08 06:59:23.589818 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.589783    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"]
Jul 08 06:59:23.597145 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.597118    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:23.599455 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.599422    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 08 06:59:23.600100 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.600073    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"]
Jul 08 06:59:23.600229 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.600111    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-t4l46\""
Jul 08 06:59:23.600229 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.600169    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 08 06:59:23.600229 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.600209    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 08 06:59:23.733121 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.733090    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64bf6568-31cf-4b84-855b-967019511798-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-rnvmv\" (UID: \"64bf6568-31cf-4b84-855b-967019511798\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:23.733299 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.733172    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64bf6568-31cf-4b84-855b-967019511798-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-rnvmv\" (UID: \"64bf6568-31cf-4b84-855b-967019511798\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:23.733299 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.733221    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bf6568-31cf-4b84-855b-967019511798-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-rnvmv\" (UID: \"64bf6568-31cf-4b84-855b-967019511798\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:23.733299 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.733247    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slglb\" (UniqueName: \"kubernetes.io/projected/64bf6568-31cf-4b84-855b-967019511798-kube-api-access-slglb\") pod \"prometheus-operator-567f7b8b7c-rnvmv\" (UID: \"64bf6568-31cf-4b84-855b-967019511798\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:23.834528 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.834489    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64bf6568-31cf-4b84-855b-967019511798-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-rnvmv\" (UID: \"64bf6568-31cf-4b84-855b-967019511798\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:23.834721 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.834575    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64bf6568-31cf-4b84-855b-967019511798-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-rnvmv\" (UID: \"64bf6568-31cf-4b84-855b-967019511798\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:23.834721 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.834632    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bf6568-31cf-4b84-855b-967019511798-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-rnvmv\" (UID: \"64bf6568-31cf-4b84-855b-967019511798\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:23.834721 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.834661    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slglb\" (UniqueName: \"kubernetes.io/projected/64bf6568-31cf-4b84-855b-967019511798-kube-api-access-slglb\") pod \"prometheus-operator-567f7b8b7c-rnvmv\" (UID: \"64bf6568-31cf-4b84-855b-967019511798\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:23.834932 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:23.834805    2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 08 06:59:23.834932 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:23.834879    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64bf6568-31cf-4b84-855b-967019511798-prometheus-operator-tls podName:64bf6568-31cf-4b84-855b-967019511798 nodeName:}" failed. No retries permitted until 2026-07-08 06:59:24.334859108 +0000 UTC m=+297.071203807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/64bf6568-31cf-4b84-855b-967019511798-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-rnvmv" (UID: "64bf6568-31cf-4b84-855b-967019511798") : secret "prometheus-operator-tls" not found
Jul 08 06:59:23.835587 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.835538    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64bf6568-31cf-4b84-855b-967019511798-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-rnvmv\" (UID: \"64bf6568-31cf-4b84-855b-967019511798\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:23.837463 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.837437    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64bf6568-31cf-4b84-855b-967019511798-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-rnvmv\" (UID: \"64bf6568-31cf-4b84-855b-967019511798\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:23.843172 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:23.843123    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slglb\" (UniqueName: \"kubernetes.io/projected/64bf6568-31cf-4b84-855b-967019511798-kube-api-access-slglb\") pod \"prometheus-operator-567f7b8b7c-rnvmv\" (UID: \"64bf6568-31cf-4b84-855b-967019511798\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:24.339640 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:24.339607    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bf6568-31cf-4b84-855b-967019511798-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-rnvmv\" (UID: \"64bf6568-31cf-4b84-855b-967019511798\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:24.342367 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:24.342343    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64bf6568-31cf-4b84-855b-967019511798-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-rnvmv\" (UID: \"64bf6568-31cf-4b84-855b-967019511798\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:24.508100 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:24.508068    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"
Jul 08 06:59:24.781627 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:24.781589    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-r4hnb" podUID="77090b45-e309-490a-973e-28c723351e73"
Jul 08 06:59:25.445031 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:25.444893    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:59:25.504426 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:25.504405    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv"]
Jul 08 06:59:25.507019 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:59:25.506993    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bf6568_31cf_4b84_855b_967019511798.slice/crio-552ba72250da6c8b6b53399fe2dd9b16625fdbbd8cbef9f08f0b7d04eececdd8 WatchSource:0}: Error finding container 552ba72250da6c8b6b53399fe2dd9b16625fdbbd8cbef9f08f0b7d04eececdd8: Status 404 returned error can't find the container with id 552ba72250da6c8b6b53399fe2dd9b16625fdbbd8cbef9f08f0b7d04eececdd8
Jul 08 06:59:25.595060 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:25.595037    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gtnj4"
Jul 08 06:59:25.638219 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:25.638148    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vbzqz" event={"ID":"c4e676c1-8726-4666-b3b7-23eb52cb32a1","Type":"ContainerStarted","Data":"fb34727186e7e5079865f42fcaf435105b8d3faacd0b08b6422ba7b6602287a0"}
Jul 08 06:59:25.639402 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:25.639379    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv" event={"ID":"64bf6568-31cf-4b84-855b-967019511798","Type":"ContainerStarted","Data":"552ba72250da6c8b6b53399fe2dd9b16625fdbbd8cbef9f08f0b7d04eececdd8"}
Jul 08 06:59:26.644177 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:26.644146    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vbzqz" event={"ID":"c4e676c1-8726-4666-b3b7-23eb52cb32a1","Type":"ContainerStarted","Data":"5e854156ae41b99b436422f8b769455ea11c7ec28ec4c9298101f10dbac9bd8a"}
Jul 08 06:59:27.650145 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:27.650100    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv" event={"ID":"64bf6568-31cf-4b84-855b-967019511798","Type":"ContainerStarted","Data":"64bf64107f21d0bd44b5ea2ba1bbfd203ac44a3bd74e3d9d2b70537bc05d9959"}
Jul 08 06:59:27.650599 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:27.650176    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv" event={"ID":"64bf6568-31cf-4b84-855b-967019511798","Type":"ContainerStarted","Data":"8bd456789fd0fe2c20e098818e8745c2ea71eeceecee4b8f6f0b13a406af9d00"}
Jul 08 06:59:27.659373 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:27.659353    2557 kubelet.go:1648] "Image garbage collection succeeded"
Jul 08 06:59:27.668845 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:27.668800    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-rnvmv" podStartSLOduration=3.506027877 podStartE2EDuration="4.668786591s" totalImagesPullingTime="1.162758714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:59:27.666920491 +0000 UTC m=+300.403265210" watchObservedRunningTime="2026-07-08 06:59:27.668786591 +0000 UTC m=+300.405131296"
Jul 08 06:59:29.661747 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.661715    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vbzqz" event={"ID":"c4e676c1-8726-4666-b3b7-23eb52cb32a1","Type":"ContainerStarted","Data":"ef73819b37522d5d372d4f11462c34a91309048238abf9c363964a268e26063d"}
Jul 08 06:59:29.696655 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.696611    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vbzqz" podStartSLOduration=33.730339024 podStartE2EDuration="40.696596025s" totalImagesPullingTime="6.966257001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 06:58:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:59:29.695680558 +0000 UTC m=+302.432025285" watchObservedRunningTime="2026-07-08 06:59:29.696596025 +0000 UTC m=+302.432940739"
Jul 08 06:59:29.966296 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.966217    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"]
Jul 08 06:59:29.969703 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.969680    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:29.976218 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.975941    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 08 06:59:29.976218 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.975974    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 08 06:59:29.976218 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.975985    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hqvbh\""
Jul 08 06:59:29.977573 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.977552    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"]
Jul 08 06:59:29.980323 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.980306    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7dzx6\" (UID: \"f258c60f-a3ae-4cce-9105-07d2d3be1ffb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:29.980409 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.980336    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7dzx6\" (UID: \"f258c60f-a3ae-4cce-9105-07d2d3be1ffb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:29.980409 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.980371    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7dzx6\" (UID: \"f258c60f-a3ae-4cce-9105-07d2d3be1ffb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:29.980480 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.980452    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p22t\" (UniqueName: \"kubernetes.io/projected/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-kube-api-access-7p22t\") pod \"openshift-state-metrics-54cf96dfcf-7dzx6\" (UID: \"f258c60f-a3ae-4cce-9105-07d2d3be1ffb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:29.981285 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.981271    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:29.983625 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.983602    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 08 06:59:29.983802 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.983786    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gq74g\""
Jul 08 06:59:29.984070 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.984050    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 08 06:59:29.984690 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.984674    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 08 06:59:29.988585 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:29.988565    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"]
Jul 08 06:59:30.005089 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.005067    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"]
Jul 08 06:59:30.007024 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.007003    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kqz2g"]
Jul 08 06:59:30.010508 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.010487    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.012574 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.012555    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 08 06:59:30.012695 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.012683    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xl7cp\""
Jul 08 06:59:30.013242 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.013227    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 08 06:59:30.013330 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.013314    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 08 06:59:30.080969 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.080941    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m65w\" (UniqueName: \"kubernetes.io/projected/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-api-access-2m65w\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.081097 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.080973    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.081097 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081002    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7dzx6\" (UID: \"f258c60f-a3ae-4cce-9105-07d2d3be1ffb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:30.081097 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081050    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-textfile\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.081220 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081102    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.081220 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081137    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7dzx6\" (UID: \"f258c60f-a3ae-4cce-9105-07d2d3be1ffb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:30.081220 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081165    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.081220 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081201    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/03a94a66-03fd-41ab-924f-2ce4f45950f5-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.081417 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081218    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03a94a66-03fd-41ab-924f-2ce4f45950f5-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.081417 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081246    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36affcb4-d312-4286-8178-fc47802561ee-sys\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.081417 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081272    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/36affcb4-d312-4286-8178-fc47802561ee-root\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.081417 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081300    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7dzx6\" (UID: \"f258c60f-a3ae-4cce-9105-07d2d3be1ffb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:30.081417 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081382    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36affcb4-d312-4286-8178-fc47802561ee-metrics-client-ca\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.081662 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:30.081389    2557 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 08 06:59:30.081662 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:30.081509    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-openshift-state-metrics-tls podName:f258c60f-a3ae-4cce-9105-07d2d3be1ffb nodeName:}" failed. No retries permitted until 2026-07-08 06:59:30.581492607 +0000 UTC m=+303.317837301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-7dzx6" (UID: "f258c60f-a3ae-4cce-9105-07d2d3be1ffb") : secret "openshift-state-metrics-tls" not found
Jul 08 06:59:30.081662 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081426    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-tls\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.081662 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081565    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p22t\" (UniqueName: \"kubernetes.io/projected/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-kube-api-access-7p22t\") pod \"openshift-state-metrics-54cf96dfcf-7dzx6\" (UID: \"f258c60f-a3ae-4cce-9105-07d2d3be1ffb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:30.081662 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081593    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.081981 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081810    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-wtmp\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.081981 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081843    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xc49\" (UniqueName: \"kubernetes.io/projected/36affcb4-d312-4286-8178-fc47802561ee-kube-api-access-6xc49\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.081981 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081872    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.081981 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.081947    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-7dzx6\" (UID: \"f258c60f-a3ae-4cce-9105-07d2d3be1ffb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:30.083410 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.083392    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-7dzx6\" (UID: \"f258c60f-a3ae-4cce-9105-07d2d3be1ffb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:30.092252 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.092226    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p22t\" (UniqueName: \"kubernetes.io/projected/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-kube-api-access-7p22t\") pod \"openshift-state-metrics-54cf96dfcf-7dzx6\" (UID: \"f258c60f-a3ae-4cce-9105-07d2d3be1ffb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:30.182351 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182323    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.182504 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182363    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.182504 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182387    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/03a94a66-03fd-41ab-924f-2ce4f45950f5-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.182504 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:30.182464    2557 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 08 06:59:30.182620 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182499    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03a94a66-03fd-41ab-924f-2ce4f45950f5-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.182620 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:30.182511    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-state-metrics-tls podName:03a94a66-03fd-41ab-924f-2ce4f45950f5 nodeName:}" failed. No retries permitted until 2026-07-08 06:59:30.682494855 +0000 UTC m=+303.418839548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-jgp8h" (UID: "03a94a66-03fd-41ab-924f-2ce4f45950f5") : secret "kube-state-metrics-tls" not found
Jul 08 06:59:30.182620 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182590    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36affcb4-d312-4286-8178-fc47802561ee-sys\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.182796 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182619    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/36affcb4-d312-4286-8178-fc47802561ee-root\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.182796 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182676    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36affcb4-d312-4286-8178-fc47802561ee-metrics-client-ca\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.182796 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182701    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36affcb4-d312-4286-8178-fc47802561ee-sys\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.182796 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182709    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-tls\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.182796 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182724    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/36affcb4-d312-4286-8178-fc47802561ee-root\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.183041 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182793    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.183041 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182827    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-wtmp\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.183041 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182853    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xc49\" (UniqueName: \"kubernetes.io/projected/36affcb4-d312-4286-8178-fc47802561ee-kube-api-access-6xc49\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.183041 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182883    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.183041 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182922    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m65w\" (UniqueName: \"kubernetes.io/projected/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-api-access-2m65w\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.183041 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182953    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.183041 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.182988    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-textfile\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.184009 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.183296    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03a94a66-03fd-41ab-924f-2ce4f45950f5-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.184009 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.183336    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36affcb4-d312-4286-8178-fc47802561ee-metrics-client-ca\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.184009 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.183462    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-wtmp\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.184009 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.183576    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.184009 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.183700    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/03a94a66-03fd-41ab-924f-2ce4f45950f5-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.184009 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.183794    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.184009 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.183838    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-textfile\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.185003 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.184983    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.185203 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.185186    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/36affcb4-d312-4286-8178-fc47802561ee-node-exporter-tls\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.185547 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.185529    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.195644 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.195625    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xc49\" (UniqueName: \"kubernetes.io/projected/36affcb4-d312-4286-8178-fc47802561ee-kube-api-access-6xc49\") pod \"node-exporter-kqz2g\" (UID: \"36affcb4-d312-4286-8178-fc47802561ee\") " pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.196248 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.196222    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m65w\" (UniqueName: \"kubernetes.io/projected/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-api-access-2m65w\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.319520 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.319492    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kqz2g"
Jul 08 06:59:30.331211 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:59:30.331187    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36affcb4_d312_4286_8178_fc47802561ee.slice/crio-4ecf2f8bcde05d0b831ddad874bfba11f0f70e25e24bef7ff3369314b59bb6d8 WatchSource:0}: Error finding container 4ecf2f8bcde05d0b831ddad874bfba11f0f70e25e24bef7ff3369314b59bb6d8: Status 404 returned error can't find the container with id 4ecf2f8bcde05d0b831ddad874bfba11f0f70e25e24bef7ff3369314b59bb6d8
Jul 08 06:59:30.332854 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.332837    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 06:59:30.586091 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.586016    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7dzx6\" (UID: \"f258c60f-a3ae-4cce-9105-07d2d3be1ffb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:30.588706 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.588684    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f258c60f-a3ae-4cce-9105-07d2d3be1ffb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-7dzx6\" (UID: \"f258c60f-a3ae-4cce-9105-07d2d3be1ffb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:30.666100 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.666071    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kqz2g" event={"ID":"36affcb4-d312-4286-8178-fc47802561ee","Type":"ContainerStarted","Data":"4ecf2f8bcde05d0b831ddad874bfba11f0f70e25e24bef7ff3369314b59bb6d8"}
Jul 08 06:59:30.686868 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.686843    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.689345 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.689317    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/03a94a66-03fd-41ab-924f-2ce4f45950f5-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-jgp8h\" (UID: \"03a94a66-03fd-41ab-924f-2ce4f45950f5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:30.880093 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.880024    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"
Jul 08 06:59:30.890167 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:30.890135    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"
Jul 08 06:59:31.050187 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:31.050154    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6"]
Jul 08 06:59:31.068422 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:31.068396    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h"]
Jul 08 06:59:31.215099 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:59:31.215028    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf258c60f_a3ae_4cce_9105_07d2d3be1ffb.slice/crio-bc5ddab8d4193aa4fc2432ad0b14672d01806cfea3df293129409bc2b357666c WatchSource:0}: Error finding container bc5ddab8d4193aa4fc2432ad0b14672d01806cfea3df293129409bc2b357666c: Status 404 returned error can't find the container with id bc5ddab8d4193aa4fc2432ad0b14672d01806cfea3df293129409bc2b357666c
Jul 08 06:59:31.215880 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:59:31.215850    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a94a66_03fd_41ab_924f_2ce4f45950f5.slice/crio-62a3f6d9f13bc611041312373db2ad438fcf8e889ebc61091799752d04893825 WatchSource:0}: Error finding container 62a3f6d9f13bc611041312373db2ad438fcf8e889ebc61091799752d04893825: Status 404 returned error can't find the container with id 62a3f6d9f13bc611041312373db2ad438fcf8e889ebc61091799752d04893825
Jul 08 06:59:31.671532 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:31.671503    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h" event={"ID":"03a94a66-03fd-41ab-924f-2ce4f45950f5","Type":"ContainerStarted","Data":"62a3f6d9f13bc611041312373db2ad438fcf8e889ebc61091799752d04893825"}
Jul 08 06:59:31.673167 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:31.673141    2557 generic.go:356] "Generic (PLEG): container finished" podID="36affcb4-d312-4286-8178-fc47802561ee" containerID="71e02f4d4c1c93d0472abdddfa041e8df288dd3e077afbaabcc6537bf48c78ab" exitCode=0
Jul 08 06:59:31.673303 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:31.673202    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kqz2g" event={"ID":"36affcb4-d312-4286-8178-fc47802561ee","Type":"ContainerDied","Data":"71e02f4d4c1c93d0472abdddfa041e8df288dd3e077afbaabcc6537bf48c78ab"}
Jul 08 06:59:31.675090 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:31.675070    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6" event={"ID":"f258c60f-a3ae-4cce-9105-07d2d3be1ffb","Type":"ContainerStarted","Data":"01cb81e40ae9a01d430afdd3892991d811ba36398c1ff21b3d5e9951eccbe90b"}
Jul 08 06:59:31.675199 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:31.675117    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6" event={"ID":"f258c60f-a3ae-4cce-9105-07d2d3be1ffb","Type":"ContainerStarted","Data":"9751326b0700962f3796cc3f17d4766b158a2d9702da03ea72e547817821d97b"}
Jul 08 06:59:31.675199 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:31.675134    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6" event={"ID":"f258c60f-a3ae-4cce-9105-07d2d3be1ffb","Type":"ContainerStarted","Data":"bc5ddab8d4193aa4fc2432ad0b14672d01806cfea3df293129409bc2b357666c"}
Jul 08 06:59:32.680134 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:32.680108    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kqz2g" event={"ID":"36affcb4-d312-4286-8178-fc47802561ee","Type":"ContainerStarted","Data":"c7f057740b5e08fb41b4912202a79f5b7cba2a128a861e56500461ef9f9e92e5"}
Jul 08 06:59:32.680419 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:32.680181    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kqz2g" event={"ID":"36affcb4-d312-4286-8178-fc47802561ee","Type":"ContainerStarted","Data":"53c752fc456e892a916ce6d9f1446b0f9c05b6364d30343691ea67d45032cc5f"}
Jul 08 06:59:32.710374 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:32.710224    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kqz2g" podStartSLOduration=2.697028843 podStartE2EDuration="3.710197629s" totalImagesPullingTime="933.645777ms" totalInitContainerRuntime="79.523009ms" isStatefulPod=true podCreationTimestamp="2026-07-08 06:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:59:32.708540439 +0000 UTC m=+305.444885155" watchObservedRunningTime="2026-07-08 06:59:32.710197629 +0000 UTC m=+305.446542340"
Jul 08 06:59:33.686321 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:33.686281    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h" event={"ID":"03a94a66-03fd-41ab-924f-2ce4f45950f5","Type":"ContainerStarted","Data":"9f02c2e6ee409904c091991c6d0602fb720de24ea843e58b8aa18641c63a8dd3"}
Jul 08 06:59:33.686728 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:33.686338    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h" event={"ID":"03a94a66-03fd-41ab-924f-2ce4f45950f5","Type":"ContainerStarted","Data":"64b25b57408795e07fbc902ba5856a4390bf6e5e119e38b1ccb4d99a66b0ebd1"}
Jul 08 06:59:33.686728 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:33.686367    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h" event={"ID":"03a94a66-03fd-41ab-924f-2ce4f45950f5","Type":"ContainerStarted","Data":"511791cc5d8a0e8ad1465e3f33fcc936513e624616172b2801ea598960807c55"}
Jul 08 06:59:33.688505 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:33.688478    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6" event={"ID":"f258c60f-a3ae-4cce-9105-07d2d3be1ffb","Type":"ContainerStarted","Data":"b8578312781483bad2fc945fee1c263b4c21a15fb74816f76f792b8ac9bd91bf"}
Jul 08 06:59:33.714078 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:33.714035    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-jgp8h" podStartSLOduration=3.282672101 podStartE2EDuration="4.714022662s" totalImagesPullingTime="1.431350561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:59:33.713825872 +0000 UTC m=+306.450170588" watchObservedRunningTime="2026-07-08 06:59:33.714022662 +0000 UTC m=+306.450367375"
Jul 08 06:59:33.732366 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:33.732329    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-7dzx6" podStartSLOduration=3.421246313 podStartE2EDuration="4.732316824s" totalImagesPullingTime="1.311070511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:59:33.731650923 +0000 UTC m=+306.467995636" watchObservedRunningTime="2026-07-08 06:59:33.732316824 +0000 UTC m=+306.468661538"
Jul 08 06:59:34.726659 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:34.726633    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf"]
Jul 08 06:59:34.732574 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:34.732553    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf"
Jul 08 06:59:34.734742 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:34.734715    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 08 06:59:34.734977 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:34.734959    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ljvv9\""
Jul 08 06:59:34.740750 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:34.740729    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf"]
Jul 08 06:59:34.822270 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:34.822224    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e6e7393-0c21-4dc5-9975-d7cba956cbae-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-thfcf\" (UID: \"8e6e7393-0c21-4dc5-9975-d7cba956cbae\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf"
Jul 08 06:59:34.822420 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:34.822297    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n45d2\" (UniqueName: \"kubernetes.io/projected/8e6e7393-0c21-4dc5-9975-d7cba956cbae-kube-api-access-n45d2\") pod \"monitoring-plugin-d5dcff7cf-thfcf\" (UID: \"8e6e7393-0c21-4dc5-9975-d7cba956cbae\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf"
Jul 08 06:59:34.923261 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:34.923232    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e6e7393-0c21-4dc5-9975-d7cba956cbae-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-thfcf\" (UID: \"8e6e7393-0c21-4dc5-9975-d7cba956cbae\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf"
Jul 08 06:59:34.923373 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:34.923275    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n45d2\" (UniqueName: \"kubernetes.io/projected/8e6e7393-0c21-4dc5-9975-d7cba956cbae-kube-api-access-n45d2\") pod \"monitoring-plugin-d5dcff7cf-thfcf\" (UID: \"8e6e7393-0c21-4dc5-9975-d7cba956cbae\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf"
Jul 08 06:59:34.923419 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:34.923378    2557 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 08 06:59:34.923465 ip-10-0-135-211 kubenswrapper[2557]: E0708 06:59:34.923435    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6e7393-0c21-4dc5-9975-d7cba956cbae-monitoring-plugin-cert podName:8e6e7393-0c21-4dc5-9975-d7cba956cbae nodeName:}" failed. No retries permitted until 2026-07-08 06:59:35.423417864 +0000 UTC m=+308.159762559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/8e6e7393-0c21-4dc5-9975-d7cba956cbae-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-thfcf" (UID: "8e6e7393-0c21-4dc5-9975-d7cba956cbae") : secret "monitoring-plugin-cert" not found
Jul 08 06:59:34.933707 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:34.933680    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n45d2\" (UniqueName: \"kubernetes.io/projected/8e6e7393-0c21-4dc5-9975-d7cba956cbae-kube-api-access-n45d2\") pod \"monitoring-plugin-d5dcff7cf-thfcf\" (UID: \"8e6e7393-0c21-4dc5-9975-d7cba956cbae\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf"
Jul 08 06:59:35.427305 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:35.427273    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e6e7393-0c21-4dc5-9975-d7cba956cbae-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-thfcf\" (UID: \"8e6e7393-0c21-4dc5-9975-d7cba956cbae\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf"
Jul 08 06:59:35.429828 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:35.429809    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8e6e7393-0c21-4dc5-9975-d7cba956cbae-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-thfcf\" (UID: \"8e6e7393-0c21-4dc5-9975-d7cba956cbae\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf"
Jul 08 06:59:35.645384 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:35.645355    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf"
Jul 08 06:59:35.776101 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:35.776029    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf"]
Jul 08 06:59:35.781448 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:59:35.781420    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6e7393_0c21_4dc5_9975_d7cba956cbae.slice/crio-454fe30a3d579569826b36c587037a408158f3c9d5d5172682d81710530560af WatchSource:0}: Error finding container 454fe30a3d579569826b36c587037a408158f3c9d5d5172682d81710530560af: Status 404 returned error can't find the container with id 454fe30a3d579569826b36c587037a408158f3c9d5d5172682d81710530560af
Jul 08 06:59:36.263276 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.262855    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 06:59:36.277232 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.277197    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.283632 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.283599    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 08 06:59:36.284147 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.283978    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 08 06:59:36.284772 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.284426    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 08 06:59:36.286059 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.286039    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 08 06:59:36.287399 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.286244    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 08 06:59:36.287399 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.286291    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 08 06:59:36.287399 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.286503    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 08 06:59:36.287399 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.286633    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 08 06:59:36.287399 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.286709    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 08 06:59:36.287399 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.286898    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 08 06:59:36.287399 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.286899    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 08 06:59:36.287399 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.287105    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l85g4\""
Jul 08 06:59:36.287399 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.287259    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 08 06:59:36.288743 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.288354    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-603d9olops51l\""
Jul 08 06:59:36.288743 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.288591    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 08 06:59:36.292004 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.291986    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 08 06:59:36.303641 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.303623    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 08 06:59:36.319720 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.319695    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336285    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336330    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336356    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6s8j\" (UniqueName: \"kubernetes.io/projected/8aa5632b-8179-49c8-970d-261913521710-kube-api-access-k6s8j\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336380    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-config\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336405    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336433    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa5632b-8179-49c8-970d-261913521710-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336456    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336478    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336503    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336526    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336552    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336587    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336617    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336640    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336668    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.336939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336695    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa5632b-8179-49c8-970d-261913521710-config-out\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.337781 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336724    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.337781 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.337781 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336826    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.337781 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.336880    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-web-config\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.437562 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.437522    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.437820 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.437573    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.437820 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.437655    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6s8j\" (UniqueName: \"kubernetes.io/projected/8aa5632b-8179-49c8-970d-261913521710-kube-api-access-k6s8j\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.437820 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.437695    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-config\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.437820 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.437719    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.437820 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.437748    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa5632b-8179-49c8-970d-261913521710-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.437820 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.437791    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.437820 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.437813    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.438297 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.437839    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.438297 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.437859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.438297 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.437888    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.438297 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.437923    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.438297 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.437953    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.438297 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.437978    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.438297 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.438007    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.438297 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.438033    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa5632b-8179-49c8-970d-261913521710-config-out\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.438297 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.438063    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.438297 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.438109    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.438297 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.438150    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.438297 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.438221    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-web-config\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.439080 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.438384    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.440545 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.440098    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.440545 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.440383    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.441882 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.441853    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.442473 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.442431    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.442980 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.442955    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.444547 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.443349    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-web-config\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.444547 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.443710    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.444547 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.444019    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.444547 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.444136    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.444547 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.444421    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.445412 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.445370    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa5632b-8179-49c8-970d-261913521710-config-out\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.447405 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.446856    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.447405 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.447352    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.447823 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.447639    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.448018 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.447980    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-config\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.448436 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.448415    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.448678 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.448517    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa5632b-8179-49c8-970d-261913521710-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.448678 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.448625    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6s8j\" (UniqueName: \"kubernetes.io/projected/8aa5632b-8179-49c8-970d-261913521710-kube-api-access-k6s8j\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.449895 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.449829    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.594899 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.594800    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:36.705508 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.705469    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf" event={"ID":"8e6e7393-0c21-4dc5-9975-d7cba956cbae","Type":"ContainerStarted","Data":"454fe30a3d579569826b36c587037a408158f3c9d5d5172682d81710530560af"}
Jul 08 06:59:36.801041 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:36.801006    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 06:59:37.085563 ip-10-0-135-211 kubenswrapper[2557]: W0708 06:59:37.085534    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa5632b_8179_49c8_970d_261913521710.slice/crio-6e3e3edc1acf4ef9cddb22a957428eb06eb1276018b23ab6905c81057001c28e WatchSource:0}: Error finding container 6e3e3edc1acf4ef9cddb22a957428eb06eb1276018b23ab6905c81057001c28e: Status 404 returned error can't find the container with id 6e3e3edc1acf4ef9cddb22a957428eb06eb1276018b23ab6905c81057001c28e
Jul 08 06:59:37.711893 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:37.711858    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf" event={"ID":"8e6e7393-0c21-4dc5-9975-d7cba956cbae","Type":"ContainerStarted","Data":"22156dead0a511fe0a94f755de2f7401e76407a2682823b4ad7c72c4cec200aa"}
Jul 08 06:59:37.712124 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:37.712092    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf"
Jul 08 06:59:37.713231 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:37.713205    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerStarted","Data":"6e3e3edc1acf4ef9cddb22a957428eb06eb1276018b23ab6905c81057001c28e"}
Jul 08 06:59:37.717941 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:37.717916    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf"
Jul 08 06:59:37.734585 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:37.734548    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-thfcf" podStartSLOduration=2.385421945 podStartE2EDuration="3.734535111s" totalImagesPullingTime="1.349113166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:59:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:59:37.734009411 +0000 UTC m=+310.470354126" watchObservedRunningTime="2026-07-08 06:59:37.734535111 +0000 UTC m=+310.470879869"
Jul 08 06:59:37.782406 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:37.782375    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 06:59:38.718144 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:38.718109    2557 generic.go:356] "Generic (PLEG): container finished" podID="8aa5632b-8179-49c8-970d-261913521710" containerID="690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3" exitCode=0
Jul 08 06:59:38.718487 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:38.718195    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerDied","Data":"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"}
Jul 08 06:59:40.458306 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:40.458266    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm" podUID="ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67" containerName="registry" containerID="cri-o://15fd5e3d99284299c52ba9a9b1d1ad4858f990d1ca411624a1c9dfca0f06a2cb" gracePeriod=30
Jul 08 06:59:40.729265 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:40.729236    2557 generic.go:356] "Generic (PLEG): container finished" podID="ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67" containerID="15fd5e3d99284299c52ba9a9b1d1ad4858f990d1ca411624a1c9dfca0f06a2cb" exitCode=0
Jul 08 06:59:40.729440 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:40.729331    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm" event={"ID":"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67","Type":"ContainerDied","Data":"15fd5e3d99284299c52ba9a9b1d1ad4858f990d1ca411624a1c9dfca0f06a2cb"}
Jul 08 06:59:41.862086 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.862067    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:59:41.888278 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.888249    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfgxz\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-kube-api-access-qfgxz\") pod \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") "
Jul 08 06:59:41.888369 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.888345    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-certificates\") pod \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") "
Jul 08 06:59:41.888433 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.888383    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-ca-trust-extracted\") pod \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") "
Jul 08 06:59:41.888494 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.888435    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-image-registry-private-configuration\") pod \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") "
Jul 08 06:59:41.888494 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.888463    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-installation-pull-secrets\") pod \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") "
Jul 08 06:59:41.888602 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.888500    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls\") pod \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") "
Jul 08 06:59:41.888602 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.888539    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-trusted-ca\") pod \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") "
Jul 08 06:59:41.888602 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.888569    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-bound-sa-token\") pod \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\" (UID: \"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67\") "
Jul 08 06:59:41.890977 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.889388    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67" (UID: "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 06:59:41.890977 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.889908    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67" (UID: "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 06:59:41.892901 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.892649    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67" (UID: "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 06:59:41.893622 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.893482    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67" (UID: "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 06:59:41.893622 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.893558    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-kube-api-access-qfgxz" (OuterVolumeSpecName: "kube-api-access-qfgxz") pod "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67" (UID: "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67"). InnerVolumeSpecName "kube-api-access-qfgxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 06:59:41.894021 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.893995    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67" (UID: "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 06:59:41.894206 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.894185    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67" (UID: "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 06:59:41.900700 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.900670    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67" (UID: "ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 06:59:41.989468 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.989386    2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-certificates\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 06:59:41.989468 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.989422    2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-ca-trust-extracted\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 06:59:41.989468 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.989436    2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-image-registry-private-configuration\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 06:59:41.989468 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.989450    2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-installation-pull-secrets\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 06:59:41.989468 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.989463    2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-registry-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 06:59:41.989965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.989476    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-trusted-ca\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 06:59:41.989965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.989488    2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-bound-sa-token\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 06:59:41.989965 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:41.989499    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfgxz\" (UniqueName: \"kubernetes.io/projected/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67-kube-api-access-qfgxz\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 06:59:42.738247 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:42.738210    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm"
Jul 08 06:59:42.738408 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:42.738207    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74bb4d45ff-fzccm" event={"ID":"ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67","Type":"ContainerDied","Data":"147f1a7c73bbcb38baaa54f38c6d9b2270acf3e5c663ae81124d801359e96bf5"}
Jul 08 06:59:42.738486 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:42.738389    2557 scope.go:117] "RemoveContainer" containerID="15fd5e3d99284299c52ba9a9b1d1ad4858f990d1ca411624a1c9dfca0f06a2cb"
Jul 08 06:59:42.741332 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:42.741303    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerStarted","Data":"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"}
Jul 08 06:59:42.741433 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:42.741373    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerStarted","Data":"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"}
Jul 08 06:59:42.761113 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:42.761090    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74bb4d45ff-fzccm"]
Jul 08 06:59:42.764247 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:42.764225    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74bb4d45ff-fzccm"]
Jul 08 06:59:43.748587 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:43.748555    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerStarted","Data":"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"}
Jul 08 06:59:43.748998 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:43.748623    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerStarted","Data":"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"}
Jul 08 06:59:43.748998 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:43.748657    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerStarted","Data":"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"}
Jul 08 06:59:43.748998 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:43.748697    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerStarted","Data":"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"}
Jul 08 06:59:43.777939 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:43.777854    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.244227096 podStartE2EDuration="7.777840333s" totalImagesPullingTime="5.514022578s" totalInitContainerRuntime="19.590659ms" isStatefulPod=false podCreationTimestamp="2026-07-08 06:59:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 06:59:43.776708409 +0000 UTC m=+316.513053125" watchObservedRunningTime="2026-07-08 06:59:43.777840333 +0000 UTC m=+316.514185046"
Jul 08 06:59:43.785306 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:43.785285    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67" path="/var/lib/kubelet/pods/ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67/volumes"
Jul 08 06:59:46.595594 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:46.595563    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 06:59:54.788411 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:54.788379    2557 generic.go:356] "Generic (PLEG): container finished" podID="1617fdda-eaa0-41c3-b37f-220f8f43c63e" containerID="8589b7f6dfb49cbf12f54fa69be3bc9466abdd2b6e50630094cabdb4b10fbd80" exitCode=0
Jul 08 06:59:54.788905 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:54.788420    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv" event={"ID":"1617fdda-eaa0-41c3-b37f-220f8f43c63e","Type":"ContainerDied","Data":"8589b7f6dfb49cbf12f54fa69be3bc9466abdd2b6e50630094cabdb4b10fbd80"}
Jul 08 06:59:54.788905 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:54.788666    2557 scope.go:117] "RemoveContainer" containerID="8589b7f6dfb49cbf12f54fa69be3bc9466abdd2b6e50630094cabdb4b10fbd80"
Jul 08 06:59:55.795696 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:55.795658    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-wpkpv" event={"ID":"1617fdda-eaa0-41c3-b37f-220f8f43c63e","Type":"ContainerStarted","Data":"c8a1aefe7d258c1323f1bda0b4e021c3c986899c17d94cde22c931841df7dde9"}
Jul 08 06:59:58.808056 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:58.808028    2557 generic.go:356] "Generic (PLEG): container finished" podID="f78e9976-f78f-4aca-a985-f2e3d54dbc64" containerID="4c6e28c52de0d510fe5eb57a7c1c749f2f26f4bfb1a0142d94a2d6c3f8ca963b" exitCode=0
Jul 08 06:59:58.808410 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:58.808075    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v" event={"ID":"f78e9976-f78f-4aca-a985-f2e3d54dbc64","Type":"ContainerDied","Data":"4c6e28c52de0d510fe5eb57a7c1c749f2f26f4bfb1a0142d94a2d6c3f8ca963b"}
Jul 08 06:59:58.808410 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:58.808360    2557 scope.go:117] "RemoveContainer" containerID="4c6e28c52de0d510fe5eb57a7c1c749f2f26f4bfb1a0142d94a2d6c3f8ca963b"
Jul 08 06:59:59.813494 ip-10-0-135-211 kubenswrapper[2557]: I0708 06:59:59.813460    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-5hz6v" event={"ID":"f78e9976-f78f-4aca-a985-f2e3d54dbc64","Type":"ContainerStarted","Data":"ae639d2b58fcbdac219fd577c77994aae52016fdea5bddea75bfcec4a6f5daee"}
Jul 08 07:00:36.595309 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:36.595265    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:36.615832 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:36.615810    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:36.969987 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:36.969916    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:41.763896 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:41.763862    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 07:00:41.766228 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:41.766204    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77090b45-e309-490a-973e-28c723351e73-metrics-certs\") pod \"network-metrics-daemon-r4hnb\" (UID: \"77090b45-e309-490a-973e-28c723351e73\") " pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 07:00:41.985577 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:41.985550    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6plfv\""
Jul 08 07:00:41.993492 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:41.993473    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r4hnb"
Jul 08 07:00:42.123014 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:42.122979    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r4hnb"]
Jul 08 07:00:42.130118 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:00:42.130091    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77090b45_e309_490a_973e_28c723351e73.slice/crio-32e068f0478d21db9eefa9a037f16d85cfa1f232cbbe095775242ef0000e589b WatchSource:0}: Error finding container 32e068f0478d21db9eefa9a037f16d85cfa1f232cbbe095775242ef0000e589b: Status 404 returned error can't find the container with id 32e068f0478d21db9eefa9a037f16d85cfa1f232cbbe095775242ef0000e589b
Jul 08 07:00:42.978675 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:42.978638    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r4hnb" event={"ID":"77090b45-e309-490a-973e-28c723351e73","Type":"ContainerStarted","Data":"32e068f0478d21db9eefa9a037f16d85cfa1f232cbbe095775242ef0000e589b"}
Jul 08 07:00:43.983808 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:43.983774    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r4hnb" event={"ID":"77090b45-e309-490a-973e-28c723351e73","Type":"ContainerStarted","Data":"7b64d53956ad1242a665130e0186790d430bd00d706c2760812c36a1dc54d62b"}
Jul 08 07:00:43.984194 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:43.983826    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r4hnb" event={"ID":"77090b45-e309-490a-973e-28c723351e73","Type":"ContainerStarted","Data":"8530d824505a16c2e2b22596a7ce0a32d3684084f65c00f1f042dae24f93ec08"}
Jul 08 07:00:44.001248 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:44.001202    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r4hnb" podStartSLOduration=375.729427709 podStartE2EDuration="6m17.00118849s" totalImagesPullingTime="1.271760781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 06:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:00:44.000103584 +0000 UTC m=+376.736448354" watchObservedRunningTime="2026-07-08 07:00:44.00118849 +0000 UTC m=+376.737533203"
Jul 08 07:00:54.647544 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.647510    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 07:00:54.647992 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.647918    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="prometheus" containerID="cri-o://bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9" gracePeriod=600
Jul 08 07:00:54.647992 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.647919    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="kube-rbac-proxy-web" containerID="cri-o://e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6" gracePeriod=600
Jul 08 07:00:54.647992 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.647932    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="thanos-sidecar" containerID="cri-o://0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107" gracePeriod=600
Jul 08 07:00:54.648158 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.647984    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="kube-rbac-proxy" containerID="cri-o://6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec" gracePeriod=600
Jul 08 07:00:54.648158 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.647920    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="kube-rbac-proxy-thanos" containerID="cri-o://235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde" gracePeriod=600
Jul 08 07:00:54.648158 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.648097    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="config-reloader" containerID="cri-o://47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263" gracePeriod=600
Jul 08 07:00:54.903856 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.903792    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:54.969816 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.969783    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-web-config\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970001 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.969831    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970001 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.969871    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-kubelet-serving-ca-bundle\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970001 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.969913    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-1\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970001 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.969959    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa5632b-8179-49c8-970d-261913521710-config-out\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970001 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.969993    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-config\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970267 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.970016    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-grpc-tls\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970267 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.970043    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa5632b-8179-49c8-970d-261913521710-tls-assets\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970267 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.970068    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-thanos-prometheus-http-client-file\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970267 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.970106    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6s8j\" (UniqueName: \"kubernetes.io/projected/8aa5632b-8179-49c8-970d-261913521710-kube-api-access-k6s8j\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970267 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.970133    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-serving-certs-ca-bundle\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970267 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.970158    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-db\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970267 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.970185    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-metrics-client-certs\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970267 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.970215    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-trusted-ca-bundle\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970267 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.970255    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970674 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.970287    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-metrics-client-ca\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970674 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.970320    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-0\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970674 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.970362    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-kube-rbac-proxy\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970674 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.970391    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-2\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.970674 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.970443    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-tls\") pod \"8aa5632b-8179-49c8-970d-261913521710\" (UID: \"8aa5632b-8179-49c8-970d-261913521710\") "
Jul 08 07:00:54.972740 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.971210    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:00:54.972740 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.972184    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:00:54.972740 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.972694    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:00:54.973043 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.972951    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:00:54.973232 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.973206    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:00:54.973332 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.973315    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:00:54.974188 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.973944    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:00:54.974188 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.974019    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-config" (OuterVolumeSpecName: "config") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:00:54.974367 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.974286    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:00:54.974705 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.974671    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:00:54.975905 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.975869    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:00:54.976282 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.976255    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:00:54.976819 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.976740    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa5632b-8179-49c8-970d-261913521710-config-out" (OuterVolumeSpecName: "config-out") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:00:54.977078 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.977043    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:00:54.977390 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.977300    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:00:54.977390 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.977319    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa5632b-8179-49c8-970d-261913521710-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:00:54.977390 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.977367    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:00:54.977627 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.977481    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:00:54.978361 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.978337    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa5632b-8179-49c8-970d-261913521710-kube-api-access-k6s8j" (OuterVolumeSpecName: "kube-api-access-k6s8j") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "kube-api-access-k6s8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:00:54.987657 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:54.987633    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-web-config" (OuterVolumeSpecName: "web-config") pod "8aa5632b-8179-49c8-970d-261913521710" (UID: "8aa5632b-8179-49c8-970d-261913521710"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:00:55.025826 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.025798    2557 generic.go:356] "Generic (PLEG): container finished" podID="8aa5632b-8179-49c8-970d-261913521710" containerID="235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde" exitCode=0
Jul 08 07:00:55.025826 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.025827    2557 generic.go:356] "Generic (PLEG): container finished" podID="8aa5632b-8179-49c8-970d-261913521710" containerID="6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec" exitCode=0
Jul 08 07:00:55.026009 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.025836    2557 generic.go:356] "Generic (PLEG): container finished" podID="8aa5632b-8179-49c8-970d-261913521710" containerID="e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6" exitCode=0
Jul 08 07:00:55.026009 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.025846    2557 generic.go:356] "Generic (PLEG): container finished" podID="8aa5632b-8179-49c8-970d-261913521710" containerID="0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107" exitCode=0
Jul 08 07:00:55.026009 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.025857    2557 generic.go:356] "Generic (PLEG): container finished" podID="8aa5632b-8179-49c8-970d-261913521710" containerID="47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263" exitCode=0
Jul 08 07:00:55.026009 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.025864    2557 generic.go:356] "Generic (PLEG): container finished" podID="8aa5632b-8179-49c8-970d-261913521710" containerID="bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9" exitCode=0
Jul 08 07:00:55.026009 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.025882    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerDied","Data":"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"}
Jul 08 07:00:55.026009 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.025893    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.026316 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.026012    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerDied","Data":"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"}
Jul 08 07:00:55.026316 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.026065    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerDied","Data":"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"}
Jul 08 07:00:55.026316 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.026077    2557 scope.go:117] "RemoveContainer" containerID="235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"
Jul 08 07:00:55.026316 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.026108    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerDied","Data":"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"}
Jul 08 07:00:55.026316 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.026148    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerDied","Data":"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"}
Jul 08 07:00:55.026316 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.026190    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerDied","Data":"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"}
Jul 08 07:00:55.026316 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.026231    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8aa5632b-8179-49c8-970d-261913521710","Type":"ContainerDied","Data":"6e3e3edc1acf4ef9cddb22a957428eb06eb1276018b23ab6905c81057001c28e"}
Jul 08 07:00:55.034135 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.034120    2557 scope.go:117] "RemoveContainer" containerID="6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"
Jul 08 07:00:55.040919 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.040902    2557 scope.go:117] "RemoveContainer" containerID="e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"
Jul 08 07:00:55.047566 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.047460    2557 scope.go:117] "RemoveContainer" containerID="0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"
Jul 08 07:00:55.050381 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.050364    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 07:00:55.055789 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.055750    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 07:00:55.055845 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.055780    2557 scope.go:117] "RemoveContainer" containerID="47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"
Jul 08 07:00:55.064124 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.062487    2557 scope.go:117] "RemoveContainer" containerID="bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"
Jul 08 07:00:55.070064 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.070049    2557 scope.go:117] "RemoveContainer" containerID="690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"
Jul 08 07:00:55.071089 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071073    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-metrics-client-ca\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071135 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071093    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071135 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071105    2557 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-kube-rbac-proxy\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071135 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071113    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071135 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071122    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071135 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071131    2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-web-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071144    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071155    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071163    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071171    2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8aa5632b-8179-49c8-970d-261913521710-config-out\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071179    2557 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071188    2557 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-grpc-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071195    2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8aa5632b-8179-49c8-970d-261913521710-tls-assets\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071205    2557 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071219    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6s8j\" (UniqueName: \"kubernetes.io/projected/8aa5632b-8179-49c8-970d-261913521710-kube-api-access-k6s8j\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071232    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071243    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8aa5632b-8179-49c8-970d-261913521710-prometheus-k8s-db\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071254    2557 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-metrics-client-certs\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071263    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aa5632b-8179-49c8-970d-261913521710-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.071301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.071271    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8aa5632b-8179-49c8-970d-261913521710-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:00:55.077223 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.077204    2557 scope.go:117] "RemoveContainer" containerID="235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"
Jul 08 07:00:55.077476 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:00:55.077455    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde\": container with ID starting with 235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde not found: ID does not exist" containerID="235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"
Jul 08 07:00:55.077521 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.077484    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"} err="failed to get container status \"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde\": rpc error: code = NotFound desc = could not find container \"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde\": container with ID starting with 235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde not found: ID does not exist"
Jul 08 07:00:55.077521 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.077520    2557 scope.go:117] "RemoveContainer" containerID="6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"
Jul 08 07:00:55.077743 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:00:55.077724    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec\": container with ID starting with 6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec not found: ID does not exist" containerID="6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"
Jul 08 07:00:55.077799 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.077749    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"} err="failed to get container status \"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec\": rpc error: code = NotFound desc = could not find container \"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec\": container with ID starting with 6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec not found: ID does not exist"
Jul 08 07:00:55.077799 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.077790    2557 scope.go:117] "RemoveContainer" containerID="e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"
Jul 08 07:00:55.078007 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:00:55.077992    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6\": container with ID starting with e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6 not found: ID does not exist" containerID="e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"
Jul 08 07:00:55.078049 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.078012    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"} err="failed to get container status \"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6\": rpc error: code = NotFound desc = could not find container \"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6\": container with ID starting with e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6 not found: ID does not exist"
Jul 08 07:00:55.078049 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.078026    2557 scope.go:117] "RemoveContainer" containerID="0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"
Jul 08 07:00:55.078209 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:00:55.078195    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107\": container with ID starting with 0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107 not found: ID does not exist" containerID="0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"
Jul 08 07:00:55.078256 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.078214    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"} err="failed to get container status \"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107\": rpc error: code = NotFound desc = could not find container \"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107\": container with ID starting with 0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107 not found: ID does not exist"
Jul 08 07:00:55.078256 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.078228    2557 scope.go:117] "RemoveContainer" containerID="47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"
Jul 08 07:00:55.078427 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:00:55.078410    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263\": container with ID starting with 47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263 not found: ID does not exist" containerID="47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"
Jul 08 07:00:55.078465 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.078431    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"} err="failed to get container status \"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263\": rpc error: code = NotFound desc = could not find container \"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263\": container with ID starting with 47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263 not found: ID does not exist"
Jul 08 07:00:55.078465 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.078444    2557 scope.go:117] "RemoveContainer" containerID="bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"
Jul 08 07:00:55.078622 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:00:55.078608    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9\": container with ID starting with bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9 not found: ID does not exist" containerID="bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"
Jul 08 07:00:55.078661 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.078625    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"} err="failed to get container status \"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9\": rpc error: code = NotFound desc = could not find container \"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9\": container with ID starting with bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9 not found: ID does not exist"
Jul 08 07:00:55.078661 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.078635    2557 scope.go:117] "RemoveContainer" containerID="690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"
Jul 08 07:00:55.078838 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:00:55.078824    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3\": container with ID starting with 690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3 not found: ID does not exist" containerID="690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"
Jul 08 07:00:55.078878 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.078841    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"} err="failed to get container status \"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3\": rpc error: code = NotFound desc = could not find container \"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3\": container with ID starting with 690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3 not found: ID does not exist"
Jul 08 07:00:55.078878 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.078854    2557 scope.go:117] "RemoveContainer" containerID="235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"
Jul 08 07:00:55.079057 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.079038    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"} err="failed to get container status \"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde\": rpc error: code = NotFound desc = could not find container \"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde\": container with ID starting with 235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde not found: ID does not exist"
Jul 08 07:00:55.079100 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.079058    2557 scope.go:117] "RemoveContainer" containerID="6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"
Jul 08 07:00:55.079275 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.079258    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"} err="failed to get container status \"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec\": rpc error: code = NotFound desc = could not find container \"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec\": container with ID starting with 6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec not found: ID does not exist"
Jul 08 07:00:55.079320 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.079278    2557 scope.go:117] "RemoveContainer" containerID="e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"
Jul 08 07:00:55.079495 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.079471    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"} err="failed to get container status \"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6\": rpc error: code = NotFound desc = could not find container \"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6\": container with ID starting with e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6 not found: ID does not exist"
Jul 08 07:00:55.079495 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.079489    2557 scope.go:117] "RemoveContainer" containerID="0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"
Jul 08 07:00:55.079669 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.079653    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"} err="failed to get container status \"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107\": rpc error: code = NotFound desc = could not find container \"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107\": container with ID starting with 0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107 not found: ID does not exist"
Jul 08 07:00:55.079711 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.079670    2557 scope.go:117] "RemoveContainer" containerID="47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"
Jul 08 07:00:55.079890 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.079876    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"} err="failed to get container status \"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263\": rpc error: code = NotFound desc = could not find container \"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263\": container with ID starting with 47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263 not found: ID does not exist"
Jul 08 07:00:55.079928 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.079890    2557 scope.go:117] "RemoveContainer" containerID="bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"
Jul 08 07:00:55.080066 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.080052    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"} err="failed to get container status \"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9\": rpc error: code = NotFound desc = could not find container \"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9\": container with ID starting with bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9 not found: ID does not exist"
Jul 08 07:00:55.080107 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.080067    2557 scope.go:117] "RemoveContainer" containerID="690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"
Jul 08 07:00:55.080239 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.080222    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"} err="failed to get container status \"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3\": rpc error: code = NotFound desc = could not find container \"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3\": container with ID starting with 690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3 not found: ID does not exist"
Jul 08 07:00:55.080281 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.080241    2557 scope.go:117] "RemoveContainer" containerID="235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"
Jul 08 07:00:55.080427 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.080412    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"} err="failed to get container status \"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde\": rpc error: code = NotFound desc = could not find container \"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde\": container with ID starting with 235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde not found: ID does not exist"
Jul 08 07:00:55.080468 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.080427    2557 scope.go:117] "RemoveContainer" containerID="6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"
Jul 08 07:00:55.080632 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.080616    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"} err="failed to get container status \"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec\": rpc error: code = NotFound desc = could not find container \"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec\": container with ID starting with 6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec not found: ID does not exist"
Jul 08 07:00:55.080632 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.080630    2557 scope.go:117] "RemoveContainer" containerID="e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"
Jul 08 07:00:55.080847 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.080831    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"} err="failed to get container status \"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6\": rpc error: code = NotFound desc = could not find container \"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6\": container with ID starting with e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6 not found: ID does not exist"
Jul 08 07:00:55.080847 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.080846    2557 scope.go:117] "RemoveContainer" containerID="0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"
Jul 08 07:00:55.081034 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.081020    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"} err="failed to get container status \"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107\": rpc error: code = NotFound desc = could not find container \"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107\": container with ID starting with 0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107 not found: ID does not exist"
Jul 08 07:00:55.081076 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.081034    2557 scope.go:117] "RemoveContainer" containerID="47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"
Jul 08 07:00:55.081202 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.081187    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"} err="failed to get container status \"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263\": rpc error: code = NotFound desc = could not find container \"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263\": container with ID starting with 47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263 not found: ID does not exist"
Jul 08 07:00:55.081238 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.081201    2557 scope.go:117] "RemoveContainer" containerID="bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"
Jul 08 07:00:55.081382 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.081367    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"} err="failed to get container status \"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9\": rpc error: code = NotFound desc = could not find container \"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9\": container with ID starting with bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9 not found: ID does not exist"
Jul 08 07:00:55.081444 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.081382    2557 scope.go:117] "RemoveContainer" containerID="690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"
Jul 08 07:00:55.081598 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.081578    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"} err="failed to get container status \"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3\": rpc error: code = NotFound desc = could not find container \"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3\": container with ID starting with 690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3 not found: ID does not exist"
Jul 08 07:00:55.081645 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.081599    2557 scope.go:117] "RemoveContainer" containerID="235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"
Jul 08 07:00:55.081825 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.081810    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"} err="failed to get container status \"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde\": rpc error: code = NotFound desc = could not find container \"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde\": container with ID starting with 235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde not found: ID does not exist"
Jul 08 07:00:55.081881 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.081824    2557 scope.go:117] "RemoveContainer" containerID="6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"
Jul 08 07:00:55.082022 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.082006    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"} err="failed to get container status \"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec\": rpc error: code = NotFound desc = could not find container \"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec\": container with ID starting with 6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec not found: ID does not exist"
Jul 08 07:00:55.082022 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.082020    2557 scope.go:117] "RemoveContainer" containerID="e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"
Jul 08 07:00:55.082212 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.082195    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"} err="failed to get container status \"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6\": rpc error: code = NotFound desc = could not find container \"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6\": container with ID starting with e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6 not found: ID does not exist"
Jul 08 07:00:55.082212 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.082211    2557 scope.go:117] "RemoveContainer" containerID="0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"
Jul 08 07:00:55.082392 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.082378    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"} err="failed to get container status \"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107\": rpc error: code = NotFound desc = could not find container \"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107\": container with ID starting with 0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107 not found: ID does not exist"
Jul 08 07:00:55.082436 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.082392    2557 scope.go:117] "RemoveContainer" containerID="47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"
Jul 08 07:00:55.082549 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.082529    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"} err="failed to get container status \"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263\": rpc error: code = NotFound desc = could not find container \"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263\": container with ID starting with 47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263 not found: ID does not exist"
Jul 08 07:00:55.082592 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.082550    2557 scope.go:117] "RemoveContainer" containerID="bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"
Jul 08 07:00:55.082745 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.082728    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"} err="failed to get container status \"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9\": rpc error: code = NotFound desc = could not find container \"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9\": container with ID starting with bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9 not found: ID does not exist"
Jul 08 07:00:55.082821 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.082745    2557 scope.go:117] "RemoveContainer" containerID="690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"
Jul 08 07:00:55.082968 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.082947    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"} err="failed to get container status \"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3\": rpc error: code = NotFound desc = could not find container \"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3\": container with ID starting with 690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3 not found: ID does not exist"
Jul 08 07:00:55.083052 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.082970    2557 scope.go:117] "RemoveContainer" containerID="235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"
Jul 08 07:00:55.083233 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.083208    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"} err="failed to get container status \"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde\": rpc error: code = NotFound desc = could not find container \"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde\": container with ID starting with 235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde not found: ID does not exist"
Jul 08 07:00:55.083233 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.083228    2557 scope.go:117] "RemoveContainer" containerID="6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"
Jul 08 07:00:55.083573 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.083549    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"} err="failed to get container status \"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec\": rpc error: code = NotFound desc = could not find container \"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec\": container with ID starting with 6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec not found: ID does not exist"
Jul 08 07:00:55.083634 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.083595    2557 scope.go:117] "RemoveContainer" containerID="e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"
Jul 08 07:00:55.083959 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.083938    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"} err="failed to get container status \"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6\": rpc error: code = NotFound desc = could not find container \"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6\": container with ID starting with e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6 not found: ID does not exist"
Jul 08 07:00:55.084159 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.083959    2557 scope.go:117] "RemoveContainer" containerID="0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"
Jul 08 07:00:55.084352 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.084331    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"} err="failed to get container status \"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107\": rpc error: code = NotFound desc = could not find container \"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107\": container with ID starting with 0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107 not found: ID does not exist"
Jul 08 07:00:55.084457 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.084351    2557 scope.go:117] "RemoveContainer" containerID="47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"
Jul 08 07:00:55.084642 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.084622    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"} err="failed to get container status \"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263\": rpc error: code = NotFound desc = could not find container \"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263\": container with ID starting with 47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263 not found: ID does not exist"
Jul 08 07:00:55.084687 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.084643    2557 scope.go:117] "RemoveContainer" containerID="bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"
Jul 08 07:00:55.084931 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.084912    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"} err="failed to get container status \"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9\": rpc error: code = NotFound desc = could not find container \"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9\": container with ID starting with bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9 not found: ID does not exist"
Jul 08 07:00:55.084988 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.084932    2557 scope.go:117] "RemoveContainer" containerID="690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"
Jul 08 07:00:55.085123 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085108    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"} err="failed to get container status \"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3\": rpc error: code = NotFound desc = could not find container \"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3\": container with ID starting with 690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3 not found: ID does not exist"
Jul 08 07:00:55.085179 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085123    2557 scope.go:117] "RemoveContainer" containerID="235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"
Jul 08 07:00:55.085345 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085329    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 07:00:55.085427 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085332    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde"} err="failed to get container status \"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde\": rpc error: code = NotFound desc = could not find container \"235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde\": container with ID starting with 235ee783dd786985bf403a56a5fa7daa96f47022c85d744efc02b78a0a67fcde not found: ID does not exist"
Jul 08 07:00:55.085472 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085428    2557 scope.go:117] "RemoveContainer" containerID="6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"
Jul 08 07:00:55.085606 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085589    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec"} err="failed to get container status \"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec\": rpc error: code = NotFound desc = could not find container \"6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec\": container with ID starting with 6ee522e27c5b4c8437bd25646902fda964a833164c89316ee53676572b98b5ec not found: ID does not exist"
Jul 08 07:00:55.085606 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085602    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="kube-rbac-proxy-web"
Jul 08 07:00:55.085676 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085607    2557 scope.go:117] "RemoveContainer" containerID="e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"
Jul 08 07:00:55.085676 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085614    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="kube-rbac-proxy-web"
Jul 08 07:00:55.085676 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085624    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="kube-rbac-proxy"
Jul 08 07:00:55.085676 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085630    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="kube-rbac-proxy"
Jul 08 07:00:55.085676 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085643    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="thanos-sidecar"
Jul 08 07:00:55.085676 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085652    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="thanos-sidecar"
Jul 08 07:00:55.085676 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085660    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="prometheus"
Jul 08 07:00:55.085676 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085667    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="prometheus"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085681    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="init-config-reloader"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085689    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="init-config-reloader"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085699    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="config-reloader"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085707    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="config-reloader"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085741    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67" containerName="registry"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085749    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67" containerName="registry"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085776    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="kube-rbac-proxy-thanos"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085784    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="kube-rbac-proxy-thanos"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085852    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="kube-rbac-proxy-web"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085861    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="kube-rbac-proxy"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085867    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="kube-rbac-proxy-thanos"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085874    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba6d20c7-5880-4d8e-9c18-7ed5ad9b9e67" containerName="registry"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085880    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="config-reloader"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085884    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="thanos-sidecar"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085889    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa5632b-8179-49c8-970d-261913521710" containerName="prometheus"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085849    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6"} err="failed to get container status \"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6\": rpc error: code = NotFound desc = could not find container \"e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6\": container with ID starting with e8ae201be5add36ca381a4793531a1152248e69d83b44a892672ebf50c7b6bb6 not found: ID does not exist"
Jul 08 07:00:55.085929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.085930    2557 scope.go:117] "RemoveContainer" containerID="0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"
Jul 08 07:00:55.086431 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.086139    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107"} err="failed to get container status \"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107\": rpc error: code = NotFound desc = could not find container \"0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107\": container with ID starting with 0bccd4395cf0417ed97697b0ab13dc2b9fb12a36eea545f397064b2ee0990107 not found: ID does not exist"
Jul 08 07:00:55.086431 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.086153    2557 scope.go:117] "RemoveContainer" containerID="47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"
Jul 08 07:00:55.086431 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.086382    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263"} err="failed to get container status \"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263\": rpc error: code = NotFound desc = could not find container \"47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263\": container with ID starting with 47601a085cb57daa63fcb220f4c8e9908e02d7fe911a933e0bc7c6c2c884e263 not found: ID does not exist"
Jul 08 07:00:55.086431 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.086398    2557 scope.go:117] "RemoveContainer" containerID="bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"
Jul 08 07:00:55.086624 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.086598    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9"} err="failed to get container status \"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9\": rpc error: code = NotFound desc = could not find container \"bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9\": container with ID starting with bf1ea5313247a641e2831026e68ff16fa415bf579b96161d55636bca1e3489a9 not found: ID does not exist"
Jul 08 07:00:55.086659 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.086624    2557 scope.go:117] "RemoveContainer" containerID="690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"
Jul 08 07:00:55.086902 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.086881    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3"} err="failed to get container status \"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3\": rpc error: code = NotFound desc = could not find container \"690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3\": container with ID starting with 690bf5b400ea6a029b6557716051f42d12af51b77db0392157f56ce6e0e94de3 not found: ID does not exist"
Jul 08 07:00:55.091375 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.091358    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.094818 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.094714    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 08 07:00:55.094818 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.094728    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l85g4\""
Jul 08 07:00:55.094818 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.094810    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 08 07:00:55.095083 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.094884    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 08 07:00:55.095083 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.094996    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 08 07:00:55.095083 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.095059    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 08 07:00:55.095238 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.095112    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 08 07:00:55.095238 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.095116    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 08 07:00:55.095343 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.095241    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 08 07:00:55.095343 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.095314    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 08 07:00:55.095723 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.095696    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 08 07:00:55.095882 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.095842    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 08 07:00:55.095882 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.095847    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 08 07:00:55.096075 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.096061    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-603d9olops51l\""
Jul 08 07:00:55.096368 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.096354    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 08 07:00:55.097793 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.097773    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 08 07:00:55.101354 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.101337    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 08 07:00:55.106725 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.106707    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 07:00:55.172586 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172528    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.172586 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172557    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.172586 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172576    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.172746 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172630    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.172746 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172663    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8wq8\" (UniqueName: \"kubernetes.io/projected/7d86829a-db91-4736-aba9-4810965b0ae2-kube-api-access-d8wq8\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.172746 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172713    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.172746 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172735    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.172898 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172783    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.172898 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172815    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.172898 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172850    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d86829a-db91-4736-aba9-4810965b0ae2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.172898 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172876    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.173013 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172906    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.173013 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172923    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.173013 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172965    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.173013 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.172989    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.173013 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.173005    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-web-config\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.173152 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.173023    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.173152 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.173038    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-config\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.173152 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.173053    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.173152 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.173068    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d86829a-db91-4736-aba9-4810965b0ae2-config-out\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.273900 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.273883    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.273985 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.273908    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.273985 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.273926    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.273985 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.273941    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8wq8\" (UniqueName: \"kubernetes.io/projected/7d86829a-db91-4736-aba9-4810965b0ae2-kube-api-access-d8wq8\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.274093 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.274064    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.274145 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.274108    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.274145 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.274132    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.274251 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.274158    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.274251 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.274183    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d86829a-db91-4736-aba9-4810965b0ae2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.274251 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.274213    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.274393 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.274262    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.274393 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.274285    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.274393 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.274310    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.274393 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.274332    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.274393 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.274353    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-web-config\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.274707 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.274655    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.275036 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.275013    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.275100 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.275073    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.275156 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.275106    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-config\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.275156 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.275134    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.275254 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.275158    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d86829a-db91-4736-aba9-4810965b0ae2-config-out\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.275254 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.275204    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.275479 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.275453    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.275858 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.275834    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.276497 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.276465    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.276752 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.276731    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.276950 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.276928    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.278598 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.277636    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.278598 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.277665    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.278598 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.278063    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.278598 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.278257    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.278598 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.278492    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.278598 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.278564    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d86829a-db91-4736-aba9-4810965b0ae2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.279120 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.279101    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.279186 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.279159    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-web-config\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.279367 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.279348    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-config\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.279553 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.279536    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7d86829a-db91-4736-aba9-4810965b0ae2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.279789 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.279746    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d86829a-db91-4736-aba9-4810965b0ae2-config-out\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.280591 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.280576    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7d86829a-db91-4736-aba9-4810965b0ae2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.282251 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.282222    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8wq8\" (UniqueName: \"kubernetes.io/projected/7d86829a-db91-4736-aba9-4810965b0ae2-kube-api-access-d8wq8\") pod \"prometheus-k8s-0\" (UID: \"7d86829a-db91-4736-aba9-4810965b0ae2\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.402391 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.402367    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:00:55.546800 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.546607    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 08 07:00:55.549276 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:00:55.549249    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d86829a_db91_4736_aba9_4810965b0ae2.slice/crio-5678b9f2d10236aa28c3be68ec2f063d4c9830874f92e220e8d535eecf439c38 WatchSource:0}: Error finding container 5678b9f2d10236aa28c3be68ec2f063d4c9830874f92e220e8d535eecf439c38: Status 404 returned error can't find the container with id 5678b9f2d10236aa28c3be68ec2f063d4c9830874f92e220e8d535eecf439c38
Jul 08 07:00:55.785900 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:55.785869    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aa5632b-8179-49c8-970d-261913521710" path="/var/lib/kubelet/pods/8aa5632b-8179-49c8-970d-261913521710/volumes"
Jul 08 07:00:56.031621 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:56.031592    2557 generic.go:356] "Generic (PLEG): container finished" podID="7d86829a-db91-4736-aba9-4810965b0ae2" containerID="d1a2510cefe51c901ae4670ed84270ae0816b0799211f37028419a4804fb7458" exitCode=0
Jul 08 07:00:56.031783 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:56.031641    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d86829a-db91-4736-aba9-4810965b0ae2","Type":"ContainerDied","Data":"d1a2510cefe51c901ae4670ed84270ae0816b0799211f37028419a4804fb7458"}
Jul 08 07:00:56.031783 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:56.031700    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d86829a-db91-4736-aba9-4810965b0ae2","Type":"ContainerStarted","Data":"5678b9f2d10236aa28c3be68ec2f063d4c9830874f92e220e8d535eecf439c38"}
Jul 08 07:00:57.039964 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:57.039926    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d86829a-db91-4736-aba9-4810965b0ae2","Type":"ContainerStarted","Data":"08cb34bb5665bb1a779e404419e922fe2a7bb48d4136890dce23e975ce770a80"}
Jul 08 07:00:57.040319 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:57.039989    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d86829a-db91-4736-aba9-4810965b0ae2","Type":"ContainerStarted","Data":"1e0d44a886de0987bad1ec977a699df4ed5495c4ab09e2225d8b62bbb600be1b"}
Jul 08 07:00:57.040319 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:57.040021    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d86829a-db91-4736-aba9-4810965b0ae2","Type":"ContainerStarted","Data":"6558bb23a148c11d1a3e0cce75813800e3e6f0435364b99c9f9f0f3f867e90c3"}
Jul 08 07:00:57.040319 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:57.040045    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d86829a-db91-4736-aba9-4810965b0ae2","Type":"ContainerStarted","Data":"5f95b5d2d370abd16967531a7bc006cc25566fae9a0e45cc9eef351bbdf76fd8"}
Jul 08 07:00:57.040319 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:57.040067    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d86829a-db91-4736-aba9-4810965b0ae2","Type":"ContainerStarted","Data":"e3dea9cc1ca8afad294583224677769be4d353ce6778f0fb26f81771d36d11b3"}
Jul 08 07:00:57.040319 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:57.040090    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d86829a-db91-4736-aba9-4810965b0ae2","Type":"ContainerStarted","Data":"8b9167430a2e946259ef2dd193b15fa2399b72c60aa364bb121cece1947f78d8"}
Jul 08 07:00:57.085922 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:00:57.085870    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.06552802 podStartE2EDuration="2.085851953s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.323933ms" isStatefulPod=false podCreationTimestamp="2026-07-08 07:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:00:57.079643557 +0000 UTC m=+389.815988271" watchObservedRunningTime="2026-07-08 07:00:57.085851953 +0000 UTC m=+389.822196668"
Jul 08 07:01:00.403234 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:01:00.403201    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:01:55.403414 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:01:55.403388    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:01:55.419092 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:01:55.419061    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:01:56.296147 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:01:56.296120    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 08 07:06:10.410385 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.410355    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-v5bth"]
Jul 08 07:06:10.412788 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.412770    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-v5bth"
Jul 08 07:06:10.415311 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.415290    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 08 07:06:10.415413 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.415311    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-rn4nm\""
Jul 08 07:06:10.415413 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.415317    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 08 07:06:10.416191 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.416173    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 08 07:06:10.422055 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.422037    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-v5bth"]
Jul 08 07:06:10.477393 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.477370    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvxt9\" (UniqueName: \"kubernetes.io/projected/cfe911c4-e4c8-46fd-972b-c751108aaf08-kube-api-access-fvxt9\") pod \"s3-init-v5bth\" (UID: \"cfe911c4-e4c8-46fd-972b-c751108aaf08\") " pod="kserve/s3-init-v5bth"
Jul 08 07:06:10.477516 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.477414    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe911c4-e4c8-46fd-972b-c751108aaf08-model-cache\") pod \"s3-init-v5bth\" (UID: \"cfe911c4-e4c8-46fd-972b-c751108aaf08\") " pod="kserve/s3-init-v5bth"
Jul 08 07:06:10.578692 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.578655    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvxt9\" (UniqueName: \"kubernetes.io/projected/cfe911c4-e4c8-46fd-972b-c751108aaf08-kube-api-access-fvxt9\") pod \"s3-init-v5bth\" (UID: \"cfe911c4-e4c8-46fd-972b-c751108aaf08\") " pod="kserve/s3-init-v5bth"
Jul 08 07:06:10.578882 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.578702    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe911c4-e4c8-46fd-972b-c751108aaf08-model-cache\") pod \"s3-init-v5bth\" (UID: \"cfe911c4-e4c8-46fd-972b-c751108aaf08\") " pod="kserve/s3-init-v5bth"
Jul 08 07:06:10.579091 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.579073    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe911c4-e4c8-46fd-972b-c751108aaf08-model-cache\") pod \"s3-init-v5bth\" (UID: \"cfe911c4-e4c8-46fd-972b-c751108aaf08\") " pod="kserve/s3-init-v5bth"
Jul 08 07:06:10.587673 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.587649    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvxt9\" (UniqueName: \"kubernetes.io/projected/cfe911c4-e4c8-46fd-972b-c751108aaf08-kube-api-access-fvxt9\") pod \"s3-init-v5bth\" (UID: \"cfe911c4-e4c8-46fd-972b-c751108aaf08\") " pod="kserve/s3-init-v5bth"
Jul 08 07:06:10.736435 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.736409    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-v5bth"
Jul 08 07:06:10.858425 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.858401    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-v5bth"]
Jul 08 07:06:10.861160 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:06:10.861132    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfe911c4_e4c8_46fd_972b_c751108aaf08.slice/crio-80074fb1f89278fe524222b7252b34f9d1e25993a820043c200a5f850ce6311b WatchSource:0}: Error finding container 80074fb1f89278fe524222b7252b34f9d1e25993a820043c200a5f850ce6311b: Status 404 returned error can't find the container with id 80074fb1f89278fe524222b7252b34f9d1e25993a820043c200a5f850ce6311b
Jul 08 07:06:10.863024 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:10.863010    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 07:06:11.264131 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:11.264097    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-v5bth" event={"ID":"cfe911c4-e4c8-46fd-972b-c751108aaf08","Type":"ContainerStarted","Data":"80074fb1f89278fe524222b7252b34f9d1e25993a820043c200a5f850ce6311b"}
Jul 08 07:06:15.284180 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:15.284139    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-v5bth" event={"ID":"cfe911c4-e4c8-46fd-972b-c751108aaf08","Type":"ContainerStarted","Data":"df09a3c6c1cc2605d3166102067f18d5b27831db0367b9c95cbfe177bcc97f5a"}
Jul 08 07:06:17.294977 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:17.294940    2557 generic.go:356] "Generic (PLEG): container finished" podID="cfe911c4-e4c8-46fd-972b-c751108aaf08" containerID="df09a3c6c1cc2605d3166102067f18d5b27831db0367b9c95cbfe177bcc97f5a" exitCode=0
Jul 08 07:06:17.295336 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:17.294992    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-v5bth" event={"ID":"cfe911c4-e4c8-46fd-972b-c751108aaf08","Type":"ContainerDied","Data":"df09a3c6c1cc2605d3166102067f18d5b27831db0367b9c95cbfe177bcc97f5a"}
Jul 08 07:06:23.323994 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:23.323952    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-v5bth" event={"ID":"cfe911c4-e4c8-46fd-972b-c751108aaf08","Type":"ContainerStarted","Data":"5b3882c0ce19794898e5583cb9ccf8e147154437e2cec27a2994c3a3b5bbcd81"}
Jul 08 07:06:23.341640 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:06:23.341585    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-v5bth" podStartSLOduration=2.8407530579999998 podStartE2EDuration="13.341564888s" totalImagesPullingTime="9.071822504s" totalInitContainerRuntime="1.428989326s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:06:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:06:23.339154473 +0000 UTC m=+716.075499187" watchObservedRunningTime="2026-07-08 07:06:23.341564888 +0000 UTC m=+716.077909603"
Jul 08 07:07:27.570725 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:27.570693    2557 generic.go:356] "Generic (PLEG): container finished" podID="cfe911c4-e4c8-46fd-972b-c751108aaf08" containerID="5b3882c0ce19794898e5583cb9ccf8e147154437e2cec27a2994c3a3b5bbcd81" exitCode=0
Jul 08 07:07:27.571127 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:27.570782    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-v5bth" event={"ID":"cfe911c4-e4c8-46fd-972b-c751108aaf08","Type":"ContainerDied","Data":"5b3882c0ce19794898e5583cb9ccf8e147154437e2cec27a2994c3a3b5bbcd81"}
Jul 08 07:07:28.705347 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:28.705325    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-v5bth"
Jul 08 07:07:28.780366 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:28.780343    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvxt9\" (UniqueName: \"kubernetes.io/projected/cfe911c4-e4c8-46fd-972b-c751108aaf08-kube-api-access-fvxt9\") pod \"cfe911c4-e4c8-46fd-972b-c751108aaf08\" (UID: \"cfe911c4-e4c8-46fd-972b-c751108aaf08\") "
Jul 08 07:07:28.780540 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:28.780415    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe911c4-e4c8-46fd-972b-c751108aaf08-model-cache\") pod \"cfe911c4-e4c8-46fd-972b-c751108aaf08\" (UID: \"cfe911c4-e4c8-46fd-972b-c751108aaf08\") "
Jul 08 07:07:28.782630 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:28.782598    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe911c4-e4c8-46fd-972b-c751108aaf08-kube-api-access-fvxt9" (OuterVolumeSpecName: "kube-api-access-fvxt9") pod "cfe911c4-e4c8-46fd-972b-c751108aaf08" (UID: "cfe911c4-e4c8-46fd-972b-c751108aaf08"). InnerVolumeSpecName "kube-api-access-fvxt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:07:28.798803 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:28.798782    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe911c4-e4c8-46fd-972b-c751108aaf08-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfe911c4-e4c8-46fd-972b-c751108aaf08" (UID: "cfe911c4-e4c8-46fd-972b-c751108aaf08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:07:28.881232 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:28.881182    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfe911c4-e4c8-46fd-972b-c751108aaf08-model-cache\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:07:28.881232 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:28.881202    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvxt9\" (UniqueName: \"kubernetes.io/projected/cfe911c4-e4c8-46fd-972b-c751108aaf08-kube-api-access-fvxt9\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:07:29.580490 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:29.580445    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-v5bth" event={"ID":"cfe911c4-e4c8-46fd-972b-c751108aaf08","Type":"ContainerDied","Data":"80074fb1f89278fe524222b7252b34f9d1e25993a820043c200a5f850ce6311b"}
Jul 08 07:07:29.580490 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:29.580468    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-v5bth"
Jul 08 07:07:29.580490 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:29.580492    2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80074fb1f89278fe524222b7252b34f9d1e25993a820043c200a5f850ce6311b"
Jul 08 07:07:41.383059 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.383022    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"]
Jul 08 07:07:41.383408 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.383350    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe911c4-e4c8-46fd-972b-c751108aaf08" containerName="s3-init"
Jul 08 07:07:41.383408 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.383363    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe911c4-e4c8-46fd-972b-c751108aaf08" containerName="s3-init"
Jul 08 07:07:41.383408 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.383381    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe911c4-e4c8-46fd-972b-c751108aaf08" containerName="download-hf-model"
Jul 08 07:07:41.383408 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.383386    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe911c4-e4c8-46fd-972b-c751108aaf08" containerName="download-hf-model"
Jul 08 07:07:41.383536 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.383443    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfe911c4-e4c8-46fd-972b-c751108aaf08" containerName="s3-init"
Jul 08 07:07:41.386881 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.386864    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.391010 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.390988    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 07:07:41.391132 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.391032    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-0b2da-predictor-serving-cert\""
Jul 08 07:07:41.391132 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.391119    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-0b2da-kube-rbac-proxy-sar-config\""
Jul 08 07:07:41.391325 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.391308    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f4gs4\""
Jul 08 07:07:41.391707 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.391688    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 07:07:41.391824 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.391738    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 07:07:41.392130 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.392116    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 07:07:41.409894 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.409872    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"]
Jul 08 07:07:41.472513 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.472483    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e3d185-f07d-454f-81b8-5252d0d3f566-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.472652 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.472529    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e3d185-f07d-454f-81b8-5252d0d3f566-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.472652 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.472554    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e3d185-f07d-454f-81b8-5252d0d3f566-proxy-tls\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.472652 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.472601    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99e3d185-f07d-454f-81b8-5252d0d3f566-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.472652 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.472624    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n9ht\" (UniqueName: \"kubernetes.io/projected/99e3d185-f07d-454f-81b8-5252d0d3f566-kube-api-access-6n9ht\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.573025 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.572998    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e3d185-f07d-454f-81b8-5252d0d3f566-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.573182 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.573037    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e3d185-f07d-454f-81b8-5252d0d3f566-proxy-tls\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.573182 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.573084    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99e3d185-f07d-454f-81b8-5252d0d3f566-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.573182 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.573112    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n9ht\" (UniqueName: \"kubernetes.io/projected/99e3d185-f07d-454f-81b8-5252d0d3f566-kube-api-access-6n9ht\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.573182 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.573155    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e3d185-f07d-454f-81b8-5252d0d3f566-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.573600 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.573575    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e3d185-f07d-454f-81b8-5252d0d3f566-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.573873 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.573849    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e3d185-f07d-454f-81b8-5252d0d3f566-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.573977 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.573849    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99e3d185-f07d-454f-81b8-5252d0d3f566-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.575810 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.575794    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e3d185-f07d-454f-81b8-5252d0d3f566-proxy-tls\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.581148 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.581126    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n9ht\" (UniqueName: \"kubernetes.io/projected/99e3d185-f07d-454f-81b8-5252d0d3f566-kube-api-access-6n9ht\") pod \"isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.696714 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.696659    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:07:41.823276 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:41.823242    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"]
Jul 08 07:07:41.826419 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:07:41.826396    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e3d185_f07d_454f_81b8_5252d0d3f566.slice/crio-044cb5251d1791d52430e37f7be1a8b1799097f401358f1f2277d7fc83fd8e93 WatchSource:0}: Error finding container 044cb5251d1791d52430e37f7be1a8b1799097f401358f1f2277d7fc83fd8e93: Status 404 returned error can't find the container with id 044cb5251d1791d52430e37f7be1a8b1799097f401358f1f2277d7fc83fd8e93
Jul 08 07:07:42.630451 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:42.630421    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" event={"ID":"99e3d185-f07d-454f-81b8-5252d0d3f566","Type":"ContainerStarted","Data":"d86d7c1810d3887fe2fa0a4ec64230e38fb70e9ea565860e96e62e9e101695ba"}
Jul 08 07:07:42.630802 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:42.630478    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" event={"ID":"99e3d185-f07d-454f-81b8-5252d0d3f566","Type":"ContainerStarted","Data":"044cb5251d1791d52430e37f7be1a8b1799097f401358f1f2277d7fc83fd8e93"}
Jul 08 07:07:46.650067 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:46.650027    2557 generic.go:356] "Generic (PLEG): container finished" podID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerID="d86d7c1810d3887fe2fa0a4ec64230e38fb70e9ea565860e96e62e9e101695ba" exitCode=0
Jul 08 07:07:46.650437 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:07:46.650107    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" event={"ID":"99e3d185-f07d-454f-81b8-5252d0d3f566","Type":"ContainerDied","Data":"d86d7c1810d3887fe2fa0a4ec64230e38fb70e9ea565860e96e62e9e101695ba"}
Jul 08 07:08:01.730891 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:01.730856    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" event={"ID":"99e3d185-f07d-454f-81b8-5252d0d3f566","Type":"ContainerStarted","Data":"e2cb34ded0ba8754fd0a1b09b30d5279dc6e9fbd93553a311bdab72abee62db9"}
Jul 08 07:08:03.741373 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:03.741341    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" event={"ID":"99e3d185-f07d-454f-81b8-5252d0d3f566","Type":"ContainerStarted","Data":"f351f87f7f4d3f1f987f8d4ba4849a4d7126bdaedcc22a3d02efe2254a883fab"}
Jul 08 07:08:07.759725 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:07.759694    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" event={"ID":"99e3d185-f07d-454f-81b8-5252d0d3f566","Type":"ContainerStarted","Data":"ee4508a3ceacddf3bfe76d22c4745d34469039dd93b0a250161fdd3caba77d5d"}
Jul 08 07:08:07.760082 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:07.759814    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:08:07.760082 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:07.759957    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:08:07.761064 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:07.761035    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 07:08:07.782087 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:07.782040    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podStartSLOduration=2.159358977 podStartE2EDuration="26.782022656s" totalImagesPullingTime="20.682021348s" totalInitContainerRuntime="3.940642331s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:07:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:08:07.779733808 +0000 UTC m=+820.516078534" watchObservedRunningTime="2026-07-08 07:08:07.782022656 +0000 UTC m=+820.518367371"
Jul 08 07:08:08.763905 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:08.763869    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 07:08:08.764326 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:08.763942    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:08:08.765775 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:08.765728    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:08:08.768970 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:08.768954    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:08:09.768395 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:09.768349    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 07:08:09.768819 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:09.768772    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:08:10.773462 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:10.773419    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 07:08:10.773918 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:10.773806    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:08:20.774245 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:20.774195    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 07:08:20.774649 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:20.774589    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:08:30.773779 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:30.773713    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 07:08:30.774281 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:30.774252    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:08:40.773726 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:40.773684    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 07:08:40.774287 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:40.774240    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:08:50.774334 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:50.774286    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 07:08:50.774823 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:08:50.774802    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:09:00.773558 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:00.773516    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 07:09:00.774167 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:00.774141    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:09:10.773958 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:10.773924    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:09:10.774440 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:10.774207    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:09:16.459703 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.459667    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"]
Jul 08 07:09:16.460268 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.459944    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container" containerID="cri-o://e2cb34ded0ba8754fd0a1b09b30d5279dc6e9fbd93553a311bdab72abee62db9" gracePeriod=30
Jul 08 07:09:16.460268 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.459961    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent" containerID="cri-o://ee4508a3ceacddf3bfe76d22c4745d34469039dd93b0a250161fdd3caba77d5d" gracePeriod=30
Jul 08 07:09:16.460268 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.459977    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kube-rbac-proxy" containerID="cri-o://f351f87f7f4d3f1f987f8d4ba4849a4d7126bdaedcc22a3d02efe2254a883fab" gracePeriod=30
Jul 08 07:09:16.470278 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.470257    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"]
Jul 08 07:09:16.473626 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.473608    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.475891 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.475808    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-a4509-predictor-serving-cert\""
Jul 08 07:09:16.475891 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.475838    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-a4509-kube-rbac-proxy-sar-config\""
Jul 08 07:09:16.491026 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.490952    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"]
Jul 08 07:09:16.553748 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.553726    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"]
Jul 08 07:09:16.557243 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.557227    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.559712 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.559689    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-a4509-predictor-serving-cert\""
Jul 08 07:09:16.559821 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.559752    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-a4509-kube-rbac-proxy-sar-config\""
Jul 08 07:09:16.567635 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.567616    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"]
Jul 08 07:09:16.617831 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.617789    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjlgg\" (UniqueName: \"kubernetes.io/projected/252cd5fc-01d6-444b-b394-04708459db4c-kube-api-access-hjlgg\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.617831 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.617836    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/252cd5fc-01d6-444b-b394-04708459db4c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.617976 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.617865    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/252cd5fc-01d6-444b-b394-04708459db4c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.617976 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.617957    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/252cd5fc-01d6-444b-b394-04708459db4c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.618053 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.617985    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/252cd5fc-01d6-444b-b394-04708459db4c-proxy-tls\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.719176 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.719117    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.719176 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.719146    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.719176 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.719169    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/252cd5fc-01d6-444b-b394-04708459db4c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.719333 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.719228    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/252cd5fc-01d6-444b-b394-04708459db4c-proxy-tls\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.719333 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.719312    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ee847f-88cd-45b6-854f-a0f2c419f0b7-cabundle-cert\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.719411 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.719371    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ee847f-88cd-45b6-854f-a0f2c419f0b7-proxy-tls\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.719446 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.719406    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj5xg\" (UniqueName: \"kubernetes.io/projected/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kube-api-access-jj5xg\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.719446 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.719430    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjlgg\" (UniqueName: \"kubernetes.io/projected/252cd5fc-01d6-444b-b394-04708459db4c-kube-api-access-hjlgg\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.719533 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.719447    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/252cd5fc-01d6-444b-b394-04708459db4c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.719533 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.719483    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/252cd5fc-01d6-444b-b394-04708459db4c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.719634 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.719537    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/252cd5fc-01d6-444b-b394-04708459db4c-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.720097 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.720073    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/252cd5fc-01d6-444b-b394-04708459db4c-cabundle-cert\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.720178 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.720084    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/252cd5fc-01d6-444b-b394-04708459db4c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.721681 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.721664    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/252cd5fc-01d6-444b-b394-04708459db4c-proxy-tls\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.727911 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.727889    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjlgg\" (UniqueName: \"kubernetes.io/projected/252cd5fc-01d6-444b-b394-04708459db4c-kube-api-access-hjlgg\") pod \"isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.809490 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.809462    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:16.820822 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.820800    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ee847f-88cd-45b6-854f-a0f2c419f0b7-proxy-tls\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.820908 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.820837    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj5xg\" (UniqueName: \"kubernetes.io/projected/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kube-api-access-jj5xg\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.820956 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.820913    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.820956 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:09:16.820929    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-serving-cert: secret "isvc-xgboost-graph-raw-a4509-predictor-serving-cert" not found
Jul 08 07:09:16.821035 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:09:16.820979    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ee847f-88cd-45b6-854f-a0f2c419f0b7-proxy-tls podName:24ee847f-88cd-45b6-854f-a0f2c419f0b7 nodeName:}" failed. No retries permitted until 2026-07-08 07:09:17.320965191 +0000 UTC m=+890.057309883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/24ee847f-88cd-45b6-854f-a0f2c419f0b7-proxy-tls") pod "isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" (UID: "24ee847f-88cd-45b6-854f-a0f2c419f0b7") : secret "isvc-xgboost-graph-raw-a4509-predictor-serving-cert" not found
Jul 08 07:09:16.821095 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.821024    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.821095 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.821072    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ee847f-88cd-45b6-854f-a0f2c419f0b7-cabundle-cert\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.821349 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.821327    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.821590 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.821575    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ee847f-88cd-45b6-854f-a0f2c419f0b7-cabundle-cert\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.821714 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.821696    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.829715 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.829693    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj5xg\" (UniqueName: \"kubernetes.io/projected/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kube-api-access-jj5xg\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:16.945372 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:16.945343    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"]
Jul 08 07:09:16.948102 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:09:16.948071    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252cd5fc_01d6_444b_b394_04708459db4c.slice/crio-c9f10e71938336eb7d4976b6bd410b7855f1905ea5e9d374f1511b7b715f366b WatchSource:0}: Error finding container c9f10e71938336eb7d4976b6bd410b7855f1905ea5e9d374f1511b7b715f366b: Status 404 returned error can't find the container with id c9f10e71938336eb7d4976b6bd410b7855f1905ea5e9d374f1511b7b715f366b
Jul 08 07:09:17.035660 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:17.035633    2557 generic.go:356] "Generic (PLEG): container finished" podID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerID="f351f87f7f4d3f1f987f8d4ba4849a4d7126bdaedcc22a3d02efe2254a883fab" exitCode=2
Jul 08 07:09:17.035807 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:17.035715    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" event={"ID":"99e3d185-f07d-454f-81b8-5252d0d3f566","Type":"ContainerDied","Data":"f351f87f7f4d3f1f987f8d4ba4849a4d7126bdaedcc22a3d02efe2254a883fab"}
Jul 08 07:09:17.037443 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:17.037419    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" event={"ID":"252cd5fc-01d6-444b-b394-04708459db4c","Type":"ContainerStarted","Data":"aef724813e1b1aa55731770e9792b12b1e5a508ed3bc927fbafdd8be2db28909"}
Jul 08 07:09:17.037544 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:17.037470    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" event={"ID":"252cd5fc-01d6-444b-b394-04708459db4c","Type":"ContainerStarted","Data":"c9f10e71938336eb7d4976b6bd410b7855f1905ea5e9d374f1511b7b715f366b"}
Jul 08 07:09:17.326391 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:17.326299    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ee847f-88cd-45b6-854f-a0f2c419f0b7-proxy-tls\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:17.328743 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:17.328715    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ee847f-88cd-45b6-854f-a0f2c419f0b7-proxy-tls\") pod \"isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:17.468324 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:17.468279    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:17.604211 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:17.604105    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"]
Jul 08 07:09:17.607096 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:09:17.607068    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ee847f_88cd_45b6_854f_a0f2c419f0b7.slice/crio-c405002ddb6d083dd51546c941daaf379b75a057df8a4c46933f97491946abeb WatchSource:0}: Error finding container c405002ddb6d083dd51546c941daaf379b75a057df8a4c46933f97491946abeb: Status 404 returned error can't find the container with id c405002ddb6d083dd51546c941daaf379b75a057df8a4c46933f97491946abeb
Jul 08 07:09:18.043380 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:18.043341    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" event={"ID":"24ee847f-88cd-45b6-854f-a0f2c419f0b7","Type":"ContainerStarted","Data":"01f703df74772d4974b8440f4a840cfac19a8042c40a09df996229186cc8a89d"}
Jul 08 07:09:18.043550 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:18.043410    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" event={"ID":"24ee847f-88cd-45b6-854f-a0f2c419f0b7","Type":"ContainerStarted","Data":"c405002ddb6d083dd51546c941daaf379b75a057df8a4c46933f97491946abeb"}
Jul 08 07:09:18.764432 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:18.764388    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 08 07:09:20.774095 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:20.774052    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 07:09:20.774459 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:20.774408    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:09:21.058009 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:21.057950    2557 generic.go:356] "Generic (PLEG): container finished" podID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerID="e2cb34ded0ba8754fd0a1b09b30d5279dc6e9fbd93553a311bdab72abee62db9" exitCode=0
Jul 08 07:09:21.058128 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:21.058027    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" event={"ID":"99e3d185-f07d-454f-81b8-5252d0d3f566","Type":"ContainerDied","Data":"e2cb34ded0ba8754fd0a1b09b30d5279dc6e9fbd93553a311bdab72abee62db9"}
Jul 08 07:09:21.059441 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:21.059417    2557 generic.go:356] "Generic (PLEG): container finished" podID="252cd5fc-01d6-444b-b394-04708459db4c" containerID="aef724813e1b1aa55731770e9792b12b1e5a508ed3bc927fbafdd8be2db28909" exitCode=0
Jul 08 07:09:21.059550 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:21.059469    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" event={"ID":"252cd5fc-01d6-444b-b394-04708459db4c","Type":"ContainerDied","Data":"aef724813e1b1aa55731770e9792b12b1e5a508ed3bc927fbafdd8be2db28909"}
Jul 08 07:09:22.065103 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:22.065071    2557 generic.go:356] "Generic (PLEG): container finished" podID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerID="01f703df74772d4974b8440f4a840cfac19a8042c40a09df996229186cc8a89d" exitCode=0
Jul 08 07:09:22.065569 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:22.065148    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" event={"ID":"24ee847f-88cd-45b6-854f-a0f2c419f0b7","Type":"ContainerDied","Data":"01f703df74772d4974b8440f4a840cfac19a8042c40a09df996229186cc8a89d"}
Jul 08 07:09:22.067593 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:22.067573    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" event={"ID":"252cd5fc-01d6-444b-b394-04708459db4c","Type":"ContainerStarted","Data":"07b3a9c194cbc99b87bd3e4fa24c9f4841e9874abbfbe8b6160511a738f6523e"}
Jul 08 07:09:22.067666 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:22.067625    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" event={"ID":"252cd5fc-01d6-444b-b394-04708459db4c","Type":"ContainerStarted","Data":"c74b2b14fc81339dd0aacf6e80ad3af393b64d7e283ec0ecc7c63cba1ed54c9d"}
Jul 08 07:09:22.067807 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:22.067792    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:22.102609 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:22.102571    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" podStartSLOduration=2.1272084270000002 podStartE2EDuration="6.102558481s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.975350054s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:09:22.100391649 +0000 UTC m=+894.836736376" watchObservedRunningTime="2026-07-08 07:09:22.102558481 +0000 UTC m=+894.838903196"
Jul 08 07:09:23.073668 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:23.073618    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:23.075358 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:23.075314    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 07:09:23.765399 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:23.765015    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 08 07:09:24.080377 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:24.080292    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 07:09:28.764353 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:28.764305    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 08 07:09:28.764811 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:28.764510    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:09:29.085722 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:29.085639    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:09:29.086320 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:29.086287    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 07:09:30.774088 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:30.774051    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 07:09:30.774577 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:30.774346    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:09:33.764530 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:33.764490    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 08 07:09:38.764672 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:38.764630    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 08 07:09:39.086911 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:39.086834    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 07:09:40.773733 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:40.773676    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 08 07:09:40.774259 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:40.773895    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:09:40.774854 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:40.774825    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:09:40.774984 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:40.774956    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:09:43.174299 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:43.174266    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" event={"ID":"24ee847f-88cd-45b6-854f-a0f2c419f0b7","Type":"ContainerStarted","Data":"9a65ac67f91512da7d3aeba0fd438d5b03ae5e437262663a13abae30c8c16350"}
Jul 08 07:09:43.174815 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:43.174325    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" event={"ID":"24ee847f-88cd-45b6-854f-a0f2c419f0b7","Type":"ContainerStarted","Data":"697bbb891ace38393f7e8e2e2cdbfc52fced9328a28fc6924daedbfd1198346f"}
Jul 08 07:09:43.174815 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:43.174608    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:43.174815 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:43.174785    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:43.175589 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:43.175568    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 07:09:43.193902 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:43.193862    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" podStartSLOduration=3.513664821 podStartE2EDuration="27.193847277s" totalImagesPullingTime="20.096623763s" totalInitContainerRuntime="3.583558693s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:09:43.192304625 +0000 UTC m=+915.928649342" watchObservedRunningTime="2026-07-08 07:09:43.193847277 +0000 UTC m=+915.930192061"
Jul 08 07:09:43.764730 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:43.764693    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 08 07:09:44.178620 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:44.178532    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 07:09:47.136244 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.136219    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:09:47.192697 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.192670    2557 generic.go:356] "Generic (PLEG): container finished" podID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerID="ee4508a3ceacddf3bfe76d22c4745d34469039dd93b0a250161fdd3caba77d5d" exitCode=0
Jul 08 07:09:47.192864 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.192784    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" event={"ID":"99e3d185-f07d-454f-81b8-5252d0d3f566","Type":"ContainerDied","Data":"ee4508a3ceacddf3bfe76d22c4745d34469039dd93b0a250161fdd3caba77d5d"}
Jul 08 07:09:47.192864 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.192817    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"
Jul 08 07:09:47.192950 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.192927    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l" event={"ID":"99e3d185-f07d-454f-81b8-5252d0d3f566","Type":"ContainerDied","Data":"044cb5251d1791d52430e37f7be1a8b1799097f401358f1f2277d7fc83fd8e93"}
Jul 08 07:09:47.192988 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.192978    2557 scope.go:117] "RemoveContainer" containerID="ee4508a3ceacddf3bfe76d22c4745d34469039dd93b0a250161fdd3caba77d5d"
Jul 08 07:09:47.201109 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.201094    2557 scope.go:117] "RemoveContainer" containerID="f351f87f7f4d3f1f987f8d4ba4849a4d7126bdaedcc22a3d02efe2254a883fab"
Jul 08 07:09:47.208876 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.208856    2557 scope.go:117] "RemoveContainer" containerID="e2cb34ded0ba8754fd0a1b09b30d5279dc6e9fbd93553a311bdab72abee62db9"
Jul 08 07:09:47.216219 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.216202    2557 scope.go:117] "RemoveContainer" containerID="d86d7c1810d3887fe2fa0a4ec64230e38fb70e9ea565860e96e62e9e101695ba"
Jul 08 07:09:47.224251 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.224228    2557 scope.go:117] "RemoveContainer" containerID="ee4508a3ceacddf3bfe76d22c4745d34469039dd93b0a250161fdd3caba77d5d"
Jul 08 07:09:47.224511 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:09:47.224491    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4508a3ceacddf3bfe76d22c4745d34469039dd93b0a250161fdd3caba77d5d\": container with ID starting with ee4508a3ceacddf3bfe76d22c4745d34469039dd93b0a250161fdd3caba77d5d not found: ID does not exist" containerID="ee4508a3ceacddf3bfe76d22c4745d34469039dd93b0a250161fdd3caba77d5d"
Jul 08 07:09:47.224575 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.224521    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4508a3ceacddf3bfe76d22c4745d34469039dd93b0a250161fdd3caba77d5d"} err="failed to get container status \"ee4508a3ceacddf3bfe76d22c4745d34469039dd93b0a250161fdd3caba77d5d\": rpc error: code = NotFound desc = could not find container \"ee4508a3ceacddf3bfe76d22c4745d34469039dd93b0a250161fdd3caba77d5d\": container with ID starting with ee4508a3ceacddf3bfe76d22c4745d34469039dd93b0a250161fdd3caba77d5d not found: ID does not exist"
Jul 08 07:09:47.224575 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.224537    2557 scope.go:117] "RemoveContainer" containerID="f351f87f7f4d3f1f987f8d4ba4849a4d7126bdaedcc22a3d02efe2254a883fab"
Jul 08 07:09:47.224807 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:09:47.224784    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f351f87f7f4d3f1f987f8d4ba4849a4d7126bdaedcc22a3d02efe2254a883fab\": container with ID starting with f351f87f7f4d3f1f987f8d4ba4849a4d7126bdaedcc22a3d02efe2254a883fab not found: ID does not exist" containerID="f351f87f7f4d3f1f987f8d4ba4849a4d7126bdaedcc22a3d02efe2254a883fab"
Jul 08 07:09:47.224859 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.224816    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f351f87f7f4d3f1f987f8d4ba4849a4d7126bdaedcc22a3d02efe2254a883fab"} err="failed to get container status \"f351f87f7f4d3f1f987f8d4ba4849a4d7126bdaedcc22a3d02efe2254a883fab\": rpc error: code = NotFound desc = could not find container \"f351f87f7f4d3f1f987f8d4ba4849a4d7126bdaedcc22a3d02efe2254a883fab\": container with ID starting with f351f87f7f4d3f1f987f8d4ba4849a4d7126bdaedcc22a3d02efe2254a883fab not found: ID does not exist"
Jul 08 07:09:47.224859 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.224832    2557 scope.go:117] "RemoveContainer" containerID="e2cb34ded0ba8754fd0a1b09b30d5279dc6e9fbd93553a311bdab72abee62db9"
Jul 08 07:09:47.225067 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:09:47.225052    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2cb34ded0ba8754fd0a1b09b30d5279dc6e9fbd93553a311bdab72abee62db9\": container with ID starting with e2cb34ded0ba8754fd0a1b09b30d5279dc6e9fbd93553a311bdab72abee62db9 not found: ID does not exist" containerID="e2cb34ded0ba8754fd0a1b09b30d5279dc6e9fbd93553a311bdab72abee62db9"
Jul 08 07:09:47.225107 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.225071    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2cb34ded0ba8754fd0a1b09b30d5279dc6e9fbd93553a311bdab72abee62db9"} err="failed to get container status \"e2cb34ded0ba8754fd0a1b09b30d5279dc6e9fbd93553a311bdab72abee62db9\": rpc error: code = NotFound desc = could not find container \"e2cb34ded0ba8754fd0a1b09b30d5279dc6e9fbd93553a311bdab72abee62db9\": container with ID starting with e2cb34ded0ba8754fd0a1b09b30d5279dc6e9fbd93553a311bdab72abee62db9 not found: ID does not exist"
Jul 08 07:09:47.225107 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.225084    2557 scope.go:117] "RemoveContainer" containerID="d86d7c1810d3887fe2fa0a4ec64230e38fb70e9ea565860e96e62e9e101695ba"
Jul 08 07:09:47.225325 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:09:47.225310    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d86d7c1810d3887fe2fa0a4ec64230e38fb70e9ea565860e96e62e9e101695ba\": container with ID starting with d86d7c1810d3887fe2fa0a4ec64230e38fb70e9ea565860e96e62e9e101695ba not found: ID does not exist" containerID="d86d7c1810d3887fe2fa0a4ec64230e38fb70e9ea565860e96e62e9e101695ba"
Jul 08 07:09:47.225387 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.225329    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d86d7c1810d3887fe2fa0a4ec64230e38fb70e9ea565860e96e62e9e101695ba"} err="failed to get container status \"d86d7c1810d3887fe2fa0a4ec64230e38fb70e9ea565860e96e62e9e101695ba\": rpc error: code = NotFound desc = could not find container \"d86d7c1810d3887fe2fa0a4ec64230e38fb70e9ea565860e96e62e9e101695ba\": container with ID starting with d86d7c1810d3887fe2fa0a4ec64230e38fb70e9ea565860e96e62e9e101695ba not found: ID does not exist"
Jul 08 07:09:47.292896 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.292823    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e3d185-f07d-454f-81b8-5252d0d3f566-kserve-provision-location\") pod \"99e3d185-f07d-454f-81b8-5252d0d3f566\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") "
Jul 08 07:09:47.292896 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.292852    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e3d185-f07d-454f-81b8-5252d0d3f566-proxy-tls\") pod \"99e3d185-f07d-454f-81b8-5252d0d3f566\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") "
Jul 08 07:09:47.292896 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.292878    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e3d185-f07d-454f-81b8-5252d0d3f566-kube-rbac-proxy-sar-config\") pod \"99e3d185-f07d-454f-81b8-5252d0d3f566\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") "
Jul 08 07:09:47.293116 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.292917    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99e3d185-f07d-454f-81b8-5252d0d3f566-cabundle-cert\") pod \"99e3d185-f07d-454f-81b8-5252d0d3f566\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") "
Jul 08 07:09:47.293116 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.292973    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n9ht\" (UniqueName: \"kubernetes.io/projected/99e3d185-f07d-454f-81b8-5252d0d3f566-kube-api-access-6n9ht\") pod \"99e3d185-f07d-454f-81b8-5252d0d3f566\" (UID: \"99e3d185-f07d-454f-81b8-5252d0d3f566\") "
Jul 08 07:09:47.293209 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.293158    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e3d185-f07d-454f-81b8-5252d0d3f566-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99e3d185-f07d-454f-81b8-5252d0d3f566" (UID: "99e3d185-f07d-454f-81b8-5252d0d3f566"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:09:47.293260 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.293235    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e3d185-f07d-454f-81b8-5252d0d3f566-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99e3d185-f07d-454f-81b8-5252d0d3f566" (UID: "99e3d185-f07d-454f-81b8-5252d0d3f566"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:09:47.293311 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.293266    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e3d185-f07d-454f-81b8-5252d0d3f566-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "99e3d185-f07d-454f-81b8-5252d0d3f566" (UID: "99e3d185-f07d-454f-81b8-5252d0d3f566"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:09:47.295143 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.295122    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e3d185-f07d-454f-81b8-5252d0d3f566-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99e3d185-f07d-454f-81b8-5252d0d3f566" (UID: "99e3d185-f07d-454f-81b8-5252d0d3f566"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:09:47.295267 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.295250    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e3d185-f07d-454f-81b8-5252d0d3f566-kube-api-access-6n9ht" (OuterVolumeSpecName: "kube-api-access-6n9ht") pod "99e3d185-f07d-454f-81b8-5252d0d3f566" (UID: "99e3d185-f07d-454f-81b8-5252d0d3f566"). InnerVolumeSpecName "kube-api-access-6n9ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:09:47.393879 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.393846    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n9ht\" (UniqueName: \"kubernetes.io/projected/99e3d185-f07d-454f-81b8-5252d0d3f566-kube-api-access-6n9ht\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:09:47.393879 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.393874    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e3d185-f07d-454f-81b8-5252d0d3f566-kserve-provision-location\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:09:47.393879 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.393885    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e3d185-f07d-454f-81b8-5252d0d3f566-proxy-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:09:47.394111 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.393896    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e3d185-f07d-454f-81b8-5252d0d3f566-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:09:47.394111 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.393907    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99e3d185-f07d-454f-81b8-5252d0d3f566-cabundle-cert\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:09:47.517630 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.517603    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"]
Jul 08 07:09:47.522005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.521982    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-0b2da-predictor-5cfb8ff6d7-8kj2l"]
Jul 08 07:09:47.785342 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:47.785311    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" path="/var/lib/kubelet/pods/99e3d185-f07d-454f-81b8-5252d0d3f566/volumes"
Jul 08 07:09:49.086984 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:49.086952    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 07:09:49.183234 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:49.183208    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:09:49.183703 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:49.183680    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 07:09:59.086933 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:59.086895    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 07:09:59.183800 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:09:59.183743    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 07:10:09.086546 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:10:09.086509    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 07:10:09.183988 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:10:09.183951    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 07:10:19.087138 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:10:19.087097    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 07:10:19.183778 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:10:19.183721    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 07:10:29.086391 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:10:29.086349    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 07:10:29.183912 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:10:29.183871    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 07:10:39.086924 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:10:39.086897    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:10:39.184963 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:10:39.184929    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:11:06.703307 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.703217    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"]
Jul 08 07:11:06.703836 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.703446    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kserve-container" containerID="cri-o://c74b2b14fc81339dd0aacf6e80ad3af393b64d7e283ec0ecc7c63cba1ed54c9d" gracePeriod=30
Jul 08 07:11:06.703836 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.703503    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kube-rbac-proxy" containerID="cri-o://07b3a9c194cbc99b87bd3e4fa24c9f4841e9874abbfbe8b6160511a738f6523e" gracePeriod=30
Jul 08 07:11:06.768541 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.768512    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"]
Jul 08 07:11:06.768888 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.768876    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container"
Jul 08 07:11:06.768935 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.768891    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container"
Jul 08 07:11:06.768935 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.768909    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kube-rbac-proxy"
Jul 08 07:11:06.768935 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.768916    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kube-rbac-proxy"
Jul 08 07:11:06.769054 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.768947    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="storage-initializer"
Jul 08 07:11:06.769054 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.768955    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="storage-initializer"
Jul 08 07:11:06.769054 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.768963    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent"
Jul 08 07:11:06.769054 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.768968    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent"
Jul 08 07:11:06.769054 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.769037    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="agent"
Jul 08 07:11:06.769054 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.769050    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kube-rbac-proxy"
Jul 08 07:11:06.769268 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.769059    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e3d185-f07d-454f-81b8-5252d0d3f566" containerName="kserve-container"
Jul 08 07:11:06.771170 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.771155    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:06.773625 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.773608    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-aea5a-kube-rbac-proxy-sar-config\""
Jul 08 07:11:06.773625 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.773620    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-aea5a-predictor-serving-cert\""
Jul 08 07:11:06.783691 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.783670    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"]
Jul 08 07:11:06.855526 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.855501    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"]
Jul 08 07:11:06.858215 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.858197    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:06.860534 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.860511    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-aea5a-predictor-serving-cert\""
Jul 08 07:11:06.860912 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.860895    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-aea5a-kube-rbac-proxy-sar-config\""
Jul 08 07:11:06.871003 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.870983    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"]
Jul 08 07:11:06.892220 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.892197    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"]
Jul 08 07:11:06.892418 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.892391    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kserve-container" containerID="cri-o://697bbb891ace38393f7e8e2e2cdbfc52fced9328a28fc6924daedbfd1198346f" gracePeriod=30
Jul 08 07:11:06.892501 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.892434    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kube-rbac-proxy" containerID="cri-o://9a65ac67f91512da7d3aeba0fd438d5b03ae5e437262663a13abae30c8c16350" gracePeriod=30
Jul 08 07:11:06.908990 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.908964    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a93541e4-a124-469f-903b-c22754a2481d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:06.909077 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.909018    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a93541e4-a124-469f-903b-c22754a2481d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:06.909148 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.909126    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxqtr\" (UniqueName: \"kubernetes.io/projected/a93541e4-a124-469f-903b-c22754a2481d-kube-api-access-jxqtr\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:06.909201 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.909168    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a93541e4-a124-469f-903b-c22754a2481d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:06.909311 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:06.909282    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a93541e4-a124-469f-903b-c22754a2481d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:07.009797 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.009750    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a93541e4-a124-469f-903b-c22754a2481d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:07.009912 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.009807    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.009912 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.009837    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.009912 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.009888    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.010083 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.009958    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a93541e4-a124-469f-903b-c22754a2481d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:07.010083 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.009989    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a93541e4-a124-469f-903b-c22754a2481d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:07.010083 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.010030    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxqtr\" (UniqueName: \"kubernetes.io/projected/a93541e4-a124-469f-903b-c22754a2481d-kube-api-access-jxqtr\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:07.010083 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.010056    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a93541e4-a124-469f-903b-c22754a2481d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:07.010083 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.010083    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swpt2\" (UniqueName: \"kubernetes.io/projected/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kube-api-access-swpt2\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.010318 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.010118    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.010318 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.010170    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a93541e4-a124-469f-903b-c22754a2481d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:07.010634 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.010607    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a93541e4-a124-469f-903b-c22754a2481d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:07.010795 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.010776    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a93541e4-a124-469f-903b-c22754a2481d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:07.012476 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.012458    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a93541e4-a124-469f-903b-c22754a2481d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:07.020669 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.020648    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxqtr\" (UniqueName: \"kubernetes.io/projected/a93541e4-a124-469f-903b-c22754a2481d-kube-api-access-jxqtr\") pod \"isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:07.080869 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.080849    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:07.111116 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.111086    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swpt2\" (UniqueName: \"kubernetes.io/projected/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kube-api-access-swpt2\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.111236 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.111122    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.111236 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.111142    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.111345 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:11:07.111260    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-serving-cert: secret "isvc-xgboost-graph-raw-hpa-aea5a-predictor-serving-cert" not found
Jul 08 07:11:07.111345 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.111286    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.111345 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:11:07.111332    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-proxy-tls podName:59ade4e3-fb68-4453-95ec-6f8ac3f136a8 nodeName:}" failed. No retries permitted until 2026-07-08 07:11:07.611308684 +0000 UTC m=+1000.347653391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-proxy-tls") pod "isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" (UID: "59ade4e3-fb68-4453-95ec-6f8ac3f136a8") : secret "isvc-xgboost-graph-raw-hpa-aea5a-predictor-serving-cert" not found
Jul 08 07:11:07.111495 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.111377    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.111495 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.111470    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.111977 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.111956    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.111977 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.111973    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.119814 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.119795    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swpt2\" (UniqueName: \"kubernetes.io/projected/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kube-api-access-swpt2\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.210860 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.208801    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"]
Jul 08 07:11:07.212298 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:11:07.212270    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93541e4_a124_469f_903b_c22754a2481d.slice/crio-96c20700ea23c52bd85e144bd6d74a8cadee2a536aafc2aadc15ee472c96c075 WatchSource:0}: Error finding container 96c20700ea23c52bd85e144bd6d74a8cadee2a536aafc2aadc15ee472c96c075: Status 404 returned error can't find the container with id 96c20700ea23c52bd85e144bd6d74a8cadee2a536aafc2aadc15ee472c96c075
Jul 08 07:11:07.524192 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.524117    2557 generic.go:356] "Generic (PLEG): container finished" podID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerID="9a65ac67f91512da7d3aeba0fd438d5b03ae5e437262663a13abae30c8c16350" exitCode=2
Jul 08 07:11:07.524344 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.524184    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" event={"ID":"24ee847f-88cd-45b6-854f-a0f2c419f0b7","Type":"ContainerDied","Data":"9a65ac67f91512da7d3aeba0fd438d5b03ae5e437262663a13abae30c8c16350"}
Jul 08 07:11:07.526234 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.526208    2557 generic.go:356] "Generic (PLEG): container finished" podID="252cd5fc-01d6-444b-b394-04708459db4c" containerID="07b3a9c194cbc99b87bd3e4fa24c9f4841e9874abbfbe8b6160511a738f6523e" exitCode=2
Jul 08 07:11:07.526362 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.526279    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" event={"ID":"252cd5fc-01d6-444b-b394-04708459db4c","Type":"ContainerDied","Data":"07b3a9c194cbc99b87bd3e4fa24c9f4841e9874abbfbe8b6160511a738f6523e"}
Jul 08 07:11:07.527724 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.527703    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" event={"ID":"a93541e4-a124-469f-903b-c22754a2481d","Type":"ContainerStarted","Data":"28f81015707d961255f21eb54923dcc1894c60b34ab19c66c0a0fe4613601840"}
Jul 08 07:11:07.527852 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.527744    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" event={"ID":"a93541e4-a124-469f-903b-c22754a2481d","Type":"ContainerStarted","Data":"96c20700ea23c52bd85e144bd6d74a8cadee2a536aafc2aadc15ee472c96c075"}
Jul 08 07:11:07.615053 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.615011    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.617393 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.617374    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.769922 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.769882    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:07.902690 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:07.902666    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"]
Jul 08 07:11:07.905121 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:11:07.905100    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ade4e3_fb68_4453_95ec_6f8ac3f136a8.slice/crio-f428f4f81cab583c2e3ff3a2799b2b01c04739e8bf7ab99384f02cab821bb7cb WatchSource:0}: Error finding container f428f4f81cab583c2e3ff3a2799b2b01c04739e8bf7ab99384f02cab821bb7cb: Status 404 returned error can't find the container with id f428f4f81cab583c2e3ff3a2799b2b01c04739e8bf7ab99384f02cab821bb7cb
Jul 08 07:11:08.533879 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:08.533839    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" event={"ID":"59ade4e3-fb68-4453-95ec-6f8ac3f136a8","Type":"ContainerStarted","Data":"0a35864568dc60b939ebbba491d12cdecfcee68e34df6dc9f47870914f6e048f"}
Jul 08 07:11:08.534061 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:08.533913    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" event={"ID":"59ade4e3-fb68-4453-95ec-6f8ac3f136a8","Type":"ContainerStarted","Data":"f428f4f81cab583c2e3ff3a2799b2b01c04739e8bf7ab99384f02cab821bb7cb"}
Jul 08 07:11:09.081394 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:09.081354    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 08 07:11:09.086777 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:09.086734    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 08 07:11:09.179502 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:09.179464    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 08 07:11:09.184071 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:09.184050    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 08 07:11:10.335752 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.335729    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:11:10.441028 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.440960    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kserve-provision-location\") pod \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") "
Jul 08 07:11:10.441028 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.440990    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj5xg\" (UniqueName: \"kubernetes.io/projected/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kube-api-access-jj5xg\") pod \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") "
Jul 08 07:11:10.441028 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.441025    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kube-rbac-proxy-sar-config\") pod \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") "
Jul 08 07:11:10.441272 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.441096    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ee847f-88cd-45b6-854f-a0f2c419f0b7-cabundle-cert\") pod \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") "
Jul 08 07:11:10.441272 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.441120    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ee847f-88cd-45b6-854f-a0f2c419f0b7-proxy-tls\") pod \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\" (UID: \"24ee847f-88cd-45b6-854f-a0f2c419f0b7\") "
Jul 08 07:11:10.441399 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.441366    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24ee847f-88cd-45b6-854f-a0f2c419f0b7" (UID: "24ee847f-88cd-45b6-854f-a0f2c419f0b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:11:10.441399 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.441387    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "24ee847f-88cd-45b6-854f-a0f2c419f0b7" (UID: "24ee847f-88cd-45b6-854f-a0f2c419f0b7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:11:10.441499 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.441449    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ee847f-88cd-45b6-854f-a0f2c419f0b7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "24ee847f-88cd-45b6-854f-a0f2c419f0b7" (UID: "24ee847f-88cd-45b6-854f-a0f2c419f0b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:11:10.443191 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.443166    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ee847f-88cd-45b6-854f-a0f2c419f0b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24ee847f-88cd-45b6-854f-a0f2c419f0b7" (UID: "24ee847f-88cd-45b6-854f-a0f2c419f0b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:11:10.443319 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.443198    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kube-api-access-jj5xg" (OuterVolumeSpecName: "kube-api-access-jj5xg") pod "24ee847f-88cd-45b6-854f-a0f2c419f0b7" (UID: "24ee847f-88cd-45b6-854f-a0f2c419f0b7"). InnerVolumeSpecName "kube-api-access-jj5xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:11:10.541884 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.541854    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:11:10.541884 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.541879    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ee847f-88cd-45b6-854f-a0f2c419f0b7-cabundle-cert\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:11:10.541884 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.541889    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ee847f-88cd-45b6-854f-a0f2c419f0b7-proxy-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:11:10.542116 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.541897    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kserve-provision-location\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:11:10.542116 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.541905    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj5xg\" (UniqueName: \"kubernetes.io/projected/24ee847f-88cd-45b6-854f-a0f2c419f0b7-kube-api-access-jj5xg\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:11:10.544477 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.544448    2557 generic.go:356] "Generic (PLEG): container finished" podID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerID="697bbb891ace38393f7e8e2e2cdbfc52fced9328a28fc6924daedbfd1198346f" exitCode=0
Jul 08 07:11:10.544590 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.544516    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"
Jul 08 07:11:10.544590 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.544538    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" event={"ID":"24ee847f-88cd-45b6-854f-a0f2c419f0b7","Type":"ContainerDied","Data":"697bbb891ace38393f7e8e2e2cdbfc52fced9328a28fc6924daedbfd1198346f"}
Jul 08 07:11:10.544660 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.544624    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch" event={"ID":"24ee847f-88cd-45b6-854f-a0f2c419f0b7","Type":"ContainerDied","Data":"c405002ddb6d083dd51546c941daaf379b75a057df8a4c46933f97491946abeb"}
Jul 08 07:11:10.544660 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.544650    2557 scope.go:117] "RemoveContainer" containerID="9a65ac67f91512da7d3aeba0fd438d5b03ae5e437262663a13abae30c8c16350"
Jul 08 07:11:10.553494 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.553481    2557 scope.go:117] "RemoveContainer" containerID="697bbb891ace38393f7e8e2e2cdbfc52fced9328a28fc6924daedbfd1198346f"
Jul 08 07:11:10.560813 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.560796    2557 scope.go:117] "RemoveContainer" containerID="01f703df74772d4974b8440f4a840cfac19a8042c40a09df996229186cc8a89d"
Jul 08 07:11:10.566746 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.566723    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"]
Jul 08 07:11:10.569443 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.569418    2557 scope.go:117] "RemoveContainer" containerID="9a65ac67f91512da7d3aeba0fd438d5b03ae5e437262663a13abae30c8c16350"
Jul 08 07:11:10.569698 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:11:10.569678    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a65ac67f91512da7d3aeba0fd438d5b03ae5e437262663a13abae30c8c16350\": container with ID starting with 9a65ac67f91512da7d3aeba0fd438d5b03ae5e437262663a13abae30c8c16350 not found: ID does not exist" containerID="9a65ac67f91512da7d3aeba0fd438d5b03ae5e437262663a13abae30c8c16350"
Jul 08 07:11:10.569796 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.569706    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a65ac67f91512da7d3aeba0fd438d5b03ae5e437262663a13abae30c8c16350"} err="failed to get container status \"9a65ac67f91512da7d3aeba0fd438d5b03ae5e437262663a13abae30c8c16350\": rpc error: code = NotFound desc = could not find container \"9a65ac67f91512da7d3aeba0fd438d5b03ae5e437262663a13abae30c8c16350\": container with ID starting with 9a65ac67f91512da7d3aeba0fd438d5b03ae5e437262663a13abae30c8c16350 not found: ID does not exist"
Jul 08 07:11:10.569796 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.569723    2557 scope.go:117] "RemoveContainer" containerID="697bbb891ace38393f7e8e2e2cdbfc52fced9328a28fc6924daedbfd1198346f"
Jul 08 07:11:10.569994 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.569976    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-a4509-predictor-67c9b5b5bf-2dxch"]
Jul 08 07:11:10.570048 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:11:10.569995    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697bbb891ace38393f7e8e2e2cdbfc52fced9328a28fc6924daedbfd1198346f\": container with ID starting with 697bbb891ace38393f7e8e2e2cdbfc52fced9328a28fc6924daedbfd1198346f not found: ID does not exist" containerID="697bbb891ace38393f7e8e2e2cdbfc52fced9328a28fc6924daedbfd1198346f"
Jul 08 07:11:10.570048 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.570021    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697bbb891ace38393f7e8e2e2cdbfc52fced9328a28fc6924daedbfd1198346f"} err="failed to get container status \"697bbb891ace38393f7e8e2e2cdbfc52fced9328a28fc6924daedbfd1198346f\": rpc error: code = NotFound desc = could not find container \"697bbb891ace38393f7e8e2e2cdbfc52fced9328a28fc6924daedbfd1198346f\": container with ID starting with 697bbb891ace38393f7e8e2e2cdbfc52fced9328a28fc6924daedbfd1198346f not found: ID does not exist"
Jul 08 07:11:10.570048 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.570036    2557 scope.go:117] "RemoveContainer" containerID="01f703df74772d4974b8440f4a840cfac19a8042c40a09df996229186cc8a89d"
Jul 08 07:11:10.570282 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:11:10.570263    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f703df74772d4974b8440f4a840cfac19a8042c40a09df996229186cc8a89d\": container with ID starting with 01f703df74772d4974b8440f4a840cfac19a8042c40a09df996229186cc8a89d not found: ID does not exist" containerID="01f703df74772d4974b8440f4a840cfac19a8042c40a09df996229186cc8a89d"
Jul 08 07:11:10.570337 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:10.570288    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f703df74772d4974b8440f4a840cfac19a8042c40a09df996229186cc8a89d"} err="failed to get container status \"01f703df74772d4974b8440f4a840cfac19a8042c40a09df996229186cc8a89d\": rpc error: code = NotFound desc = could not find container \"01f703df74772d4974b8440f4a840cfac19a8042c40a09df996229186cc8a89d\": container with ID starting with 01f703df74772d4974b8440f4a840cfac19a8042c40a09df996229186cc8a89d not found: ID does not exist"
Jul 08 07:11:11.137547 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.137525    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:11:11.247545 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.247523    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/252cd5fc-01d6-444b-b394-04708459db4c-proxy-tls\") pod \"252cd5fc-01d6-444b-b394-04708459db4c\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") "
Jul 08 07:11:11.247694 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.247565    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/252cd5fc-01d6-444b-b394-04708459db4c-cabundle-cert\") pod \"252cd5fc-01d6-444b-b394-04708459db4c\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") "
Jul 08 07:11:11.247694 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.247602    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/252cd5fc-01d6-444b-b394-04708459db4c-kube-rbac-proxy-sar-config\") pod \"252cd5fc-01d6-444b-b394-04708459db4c\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") "
Jul 08 07:11:11.247694 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.247644    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/252cd5fc-01d6-444b-b394-04708459db4c-kserve-provision-location\") pod \"252cd5fc-01d6-444b-b394-04708459db4c\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") "
Jul 08 07:11:11.247694 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.247676    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjlgg\" (UniqueName: \"kubernetes.io/projected/252cd5fc-01d6-444b-b394-04708459db4c-kube-api-access-hjlgg\") pod \"252cd5fc-01d6-444b-b394-04708459db4c\" (UID: \"252cd5fc-01d6-444b-b394-04708459db4c\") "
Jul 08 07:11:11.248008 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.247976    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252cd5fc-01d6-444b-b394-04708459db4c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "252cd5fc-01d6-444b-b394-04708459db4c" (UID: "252cd5fc-01d6-444b-b394-04708459db4c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:11:11.248102 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.248014    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252cd5fc-01d6-444b-b394-04708459db4c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "252cd5fc-01d6-444b-b394-04708459db4c" (UID: "252cd5fc-01d6-444b-b394-04708459db4c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:11:11.248102 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.248018    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252cd5fc-01d6-444b-b394-04708459db4c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "252cd5fc-01d6-444b-b394-04708459db4c" (UID: "252cd5fc-01d6-444b-b394-04708459db4c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:11:11.249789 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.249743    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/252cd5fc-01d6-444b-b394-04708459db4c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "252cd5fc-01d6-444b-b394-04708459db4c" (UID: "252cd5fc-01d6-444b-b394-04708459db4c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:11:11.249887 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.249810    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252cd5fc-01d6-444b-b394-04708459db4c-kube-api-access-hjlgg" (OuterVolumeSpecName: "kube-api-access-hjlgg") pod "252cd5fc-01d6-444b-b394-04708459db4c" (UID: "252cd5fc-01d6-444b-b394-04708459db4c"). InnerVolumeSpecName "kube-api-access-hjlgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:11:11.348809 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.348779    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/252cd5fc-01d6-444b-b394-04708459db4c-cabundle-cert\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:11:11.348809 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.348812    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/252cd5fc-01d6-444b-b394-04708459db4c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:11:11.349175 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.348830    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/252cd5fc-01d6-444b-b394-04708459db4c-kserve-provision-location\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:11:11.349175 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.348844    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjlgg\" (UniqueName: \"kubernetes.io/projected/252cd5fc-01d6-444b-b394-04708459db4c-kube-api-access-hjlgg\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:11:11.349175 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.348858    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/252cd5fc-01d6-444b-b394-04708459db4c-proxy-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:11:11.551511 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.551479    2557 generic.go:356] "Generic (PLEG): container finished" podID="252cd5fc-01d6-444b-b394-04708459db4c" containerID="c74b2b14fc81339dd0aacf6e80ad3af393b64d7e283ec0ecc7c63cba1ed54c9d" exitCode=0
Jul 08 07:11:11.551699 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.551541    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"
Jul 08 07:11:11.551699 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.551561    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" event={"ID":"252cd5fc-01d6-444b-b394-04708459db4c","Type":"ContainerDied","Data":"c74b2b14fc81339dd0aacf6e80ad3af393b64d7e283ec0ecc7c63cba1ed54c9d"}
Jul 08 07:11:11.551699 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.551615    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm" event={"ID":"252cd5fc-01d6-444b-b394-04708459db4c","Type":"ContainerDied","Data":"c9f10e71938336eb7d4976b6bd410b7855f1905ea5e9d374f1511b7b715f366b"}
Jul 08 07:11:11.551699 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.551646    2557 scope.go:117] "RemoveContainer" containerID="07b3a9c194cbc99b87bd3e4fa24c9f4841e9874abbfbe8b6160511a738f6523e"
Jul 08 07:11:11.553214 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.553188    2557 generic.go:356] "Generic (PLEG): container finished" podID="a93541e4-a124-469f-903b-c22754a2481d" containerID="28f81015707d961255f21eb54923dcc1894c60b34ab19c66c0a0fe4613601840" exitCode=0
Jul 08 07:11:11.553349 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.553260    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" event={"ID":"a93541e4-a124-469f-903b-c22754a2481d","Type":"ContainerDied","Data":"28f81015707d961255f21eb54923dcc1894c60b34ab19c66c0a0fe4613601840"}
Jul 08 07:11:11.560571 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.560553    2557 scope.go:117] "RemoveContainer" containerID="c74b2b14fc81339dd0aacf6e80ad3af393b64d7e283ec0ecc7c63cba1ed54c9d"
Jul 08 07:11:11.568396 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.568378    2557 scope.go:117] "RemoveContainer" containerID="aef724813e1b1aa55731770e9792b12b1e5a508ed3bc927fbafdd8be2db28909"
Jul 08 07:11:11.581867 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.581851    2557 scope.go:117] "RemoveContainer" containerID="07b3a9c194cbc99b87bd3e4fa24c9f4841e9874abbfbe8b6160511a738f6523e"
Jul 08 07:11:11.582137 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:11:11.582120    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b3a9c194cbc99b87bd3e4fa24c9f4841e9874abbfbe8b6160511a738f6523e\": container with ID starting with 07b3a9c194cbc99b87bd3e4fa24c9f4841e9874abbfbe8b6160511a738f6523e not found: ID does not exist" containerID="07b3a9c194cbc99b87bd3e4fa24c9f4841e9874abbfbe8b6160511a738f6523e"
Jul 08 07:11:11.582188 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.582144    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b3a9c194cbc99b87bd3e4fa24c9f4841e9874abbfbe8b6160511a738f6523e"} err="failed to get container status \"07b3a9c194cbc99b87bd3e4fa24c9f4841e9874abbfbe8b6160511a738f6523e\": rpc error: code = NotFound desc = could not find container \"07b3a9c194cbc99b87bd3e4fa24c9f4841e9874abbfbe8b6160511a738f6523e\": container with ID starting with 07b3a9c194cbc99b87bd3e4fa24c9f4841e9874abbfbe8b6160511a738f6523e not found: ID does not exist"
Jul 08 07:11:11.582188 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.582160    2557 scope.go:117] "RemoveContainer" containerID="c74b2b14fc81339dd0aacf6e80ad3af393b64d7e283ec0ecc7c63cba1ed54c9d"
Jul 08 07:11:11.582375 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:11:11.582358    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74b2b14fc81339dd0aacf6e80ad3af393b64d7e283ec0ecc7c63cba1ed54c9d\": container with ID starting with c74b2b14fc81339dd0aacf6e80ad3af393b64d7e283ec0ecc7c63cba1ed54c9d not found: ID does not exist" containerID="c74b2b14fc81339dd0aacf6e80ad3af393b64d7e283ec0ecc7c63cba1ed54c9d"
Jul 08 07:11:11.582419 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.582382    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74b2b14fc81339dd0aacf6e80ad3af393b64d7e283ec0ecc7c63cba1ed54c9d"} err="failed to get container status \"c74b2b14fc81339dd0aacf6e80ad3af393b64d7e283ec0ecc7c63cba1ed54c9d\": rpc error: code = NotFound desc = could not find container \"c74b2b14fc81339dd0aacf6e80ad3af393b64d7e283ec0ecc7c63cba1ed54c9d\": container with ID starting with c74b2b14fc81339dd0aacf6e80ad3af393b64d7e283ec0ecc7c63cba1ed54c9d not found: ID does not exist"
Jul 08 07:11:11.582419 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.582398    2557 scope.go:117] "RemoveContainer" containerID="aef724813e1b1aa55731770e9792b12b1e5a508ed3bc927fbafdd8be2db28909"
Jul 08 07:11:11.582631 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:11:11.582615    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef724813e1b1aa55731770e9792b12b1e5a508ed3bc927fbafdd8be2db28909\": container with ID starting with aef724813e1b1aa55731770e9792b12b1e5a508ed3bc927fbafdd8be2db28909 not found: ID does not exist" containerID="aef724813e1b1aa55731770e9792b12b1e5a508ed3bc927fbafdd8be2db28909"
Jul 08 07:11:11.582672 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.582643    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef724813e1b1aa55731770e9792b12b1e5a508ed3bc927fbafdd8be2db28909"} err="failed to get container status \"aef724813e1b1aa55731770e9792b12b1e5a508ed3bc927fbafdd8be2db28909\": rpc error: code = NotFound desc = could not find container \"aef724813e1b1aa55731770e9792b12b1e5a508ed3bc927fbafdd8be2db28909\": container with ID starting with aef724813e1b1aa55731770e9792b12b1e5a508ed3bc927fbafdd8be2db28909 not found: ID does not exist"
Jul 08 07:11:11.588084 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.588062    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"]
Jul 08 07:11:11.591892 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.591873    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-a4509-predictor-7966d8cf47-mg8lm"]
Jul 08 07:11:11.784925 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.784897    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" path="/var/lib/kubelet/pods/24ee847f-88cd-45b6-854f-a0f2c419f0b7/volumes"
Jul 08 07:11:11.785419 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:11.785405    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="252cd5fc-01d6-444b-b394-04708459db4c" path="/var/lib/kubelet/pods/252cd5fc-01d6-444b-b394-04708459db4c/volumes"
Jul 08 07:11:12.559287 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:12.559256    2557 generic.go:356] "Generic (PLEG): container finished" podID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerID="0a35864568dc60b939ebbba491d12cdecfcee68e34df6dc9f47870914f6e048f" exitCode=0
Jul 08 07:11:12.559654 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:12.559327    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" event={"ID":"59ade4e3-fb68-4453-95ec-6f8ac3f136a8","Type":"ContainerDied","Data":"0a35864568dc60b939ebbba491d12cdecfcee68e34df6dc9f47870914f6e048f"}
Jul 08 07:11:12.562474 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:12.562449    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" event={"ID":"a93541e4-a124-469f-903b-c22754a2481d","Type":"ContainerStarted","Data":"80f3f8957725440faef02fefcddd47104e6021d5cace233a88072f5832df07ee"}
Jul 08 07:11:12.562568 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:12.562505    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" event={"ID":"a93541e4-a124-469f-903b-c22754a2481d","Type":"ContainerStarted","Data":"f10deb6ddc15b9b10b518d62433618f58d3730a35375f1af11e673de71da0d88"}
Jul 08 07:11:12.564346 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:12.564328    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:12.564460 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:12.564379    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:12.565952 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:12.565929    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 07:11:12.596033 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:12.595982    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podStartSLOduration=2.947966778 podStartE2EDuration="6.595961384s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.647994606s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:11:12.594664145 +0000 UTC m=+1005.331008861" watchObservedRunningTime="2026-07-08 07:11:12.595961384 +0000 UTC m=+1005.332306101"
Jul 08 07:11:13.568957 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:13.568918    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" event={"ID":"59ade4e3-fb68-4453-95ec-6f8ac3f136a8","Type":"ContainerStarted","Data":"5d1177799579431046c28cbdac6b89da10d9ce31d1bce7300645ba16c20faea9"}
Jul 08 07:11:13.569414 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:13.568996    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" event={"ID":"59ade4e3-fb68-4453-95ec-6f8ac3f136a8","Type":"ContainerStarted","Data":"ba891c312923545aa11604c66185196e8be0962f11bb74992375e6b301407f0f"}
Jul 08 07:11:13.569414 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:13.569175    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 07:11:13.569532 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:13.569425    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:13.569604 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:13.569584    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:13.570239 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:13.570211    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 08 07:11:13.589274 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:13.589227    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" podStartSLOduration=3.706930199 podStartE2EDuration="7.589215924s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.882285725s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:11:13.588064519 +0000 UTC m=+1006.324409233" watchObservedRunningTime="2026-07-08 07:11:13.589215924 +0000 UTC m=+1006.325560700"
Jul 08 07:11:14.574178 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:14.574135    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 08 07:11:18.573729 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:18.573702    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:11:18.574346 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:18.574311    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 07:11:19.578599 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:19.578570    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:11:19.579087 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:19.579060    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 08 07:11:28.574332 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:28.574285    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 07:11:29.579808 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:29.579771    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 08 07:11:38.574970 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:38.574925    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 07:11:39.579151 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:39.579111    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 08 07:11:48.574228 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:48.574182    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 07:11:49.579325 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:49.579282    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 08 07:11:58.574894 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:58.574852    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 07:11:59.579488 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:11:59.579447    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 08 07:12:08.574204 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:08.574153    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 07:12:09.579638 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:09.579611    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:12:18.575147 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:18.575109    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 07:12:19.781330 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:19.781235    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 07:12:29.784560 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:29.784534    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:12:57.055096 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055054    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"]
Jul 08 07:12:57.055671 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055590    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kserve-container"
Jul 08 07:12:57.055671 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055609    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kserve-container"
Jul 08 07:12:57.055671 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055628    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kube-rbac-proxy"
Jul 08 07:12:57.055671 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055635    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kube-rbac-proxy"
Jul 08 07:12:57.055671 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055657    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kube-rbac-proxy"
Jul 08 07:12:57.055671 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055666    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kube-rbac-proxy"
Jul 08 07:12:57.056005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055684    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kserve-container"
Jul 08 07:12:57.056005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055692    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kserve-container"
Jul 08 07:12:57.056005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055702    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="storage-initializer"
Jul 08 07:12:57.056005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055712    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="storage-initializer"
Jul 08 07:12:57.056005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055721    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="storage-initializer"
Jul 08 07:12:57.056005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055729    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="storage-initializer"
Jul 08 07:12:57.056005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055832    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kube-rbac-proxy"
Jul 08 07:12:57.056005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055848    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kserve-container"
Jul 08 07:12:57.056005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055859    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="252cd5fc-01d6-444b-b394-04708459db4c" containerName="kserve-container"
Jul 08 07:12:57.056005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.055871    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ee847f-88cd-45b6-854f-a0f2c419f0b7" containerName="kube-rbac-proxy"
Jul 08 07:12:57.058953 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.058931    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:12:57.068609 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.068589    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-53e83-predictor-serving-cert\""
Jul 08 07:12:57.068721 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.068597    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-53e83-kube-rbac-proxy-sar-config\""
Jul 08 07:12:57.078485 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.078465    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"]
Jul 08 07:12:57.137945 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.137915    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"]
Jul 08 07:12:57.138490 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.138459    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container" containerID="cri-o://f10deb6ddc15b9b10b518d62433618f58d3730a35375f1af11e673de71da0d88" gracePeriod=30
Jul 08 07:12:57.141497 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.139006    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kube-rbac-proxy" containerID="cri-o://80f3f8957725440faef02fefcddd47104e6021d5cace233a88072f5832df07ee" gracePeriod=30
Jul 08 07:12:57.172621 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.172592    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4e94490-b45d-4292-baf1-0aa81f223a21-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-53e83-predictor-78879675f8-nsgwk\" (UID: \"b4e94490-b45d-4292-baf1-0aa81f223a21\") " pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:12:57.172824 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.172797    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4e94490-b45d-4292-baf1-0aa81f223a21-proxy-tls\") pod \"message-dumper-raw-53e83-predictor-78879675f8-nsgwk\" (UID: \"b4e94490-b45d-4292-baf1-0aa81f223a21\") " pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:12:57.172936 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.172847    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vt4n\" (UniqueName: \"kubernetes.io/projected/b4e94490-b45d-4292-baf1-0aa81f223a21-kube-api-access-8vt4n\") pod \"message-dumper-raw-53e83-predictor-78879675f8-nsgwk\" (UID: \"b4e94490-b45d-4292-baf1-0aa81f223a21\") " pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:12:57.194162 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.194134    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"]
Jul 08 07:12:57.194373 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.194339    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kserve-container" containerID="cri-o://ba891c312923545aa11604c66185196e8be0962f11bb74992375e6b301407f0f" gracePeriod=30
Jul 08 07:12:57.194434 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.194377    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kube-rbac-proxy" containerID="cri-o://5d1177799579431046c28cbdac6b89da10d9ce31d1bce7300645ba16c20faea9" gracePeriod=30
Jul 08 07:12:57.273528 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.273501    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4e94490-b45d-4292-baf1-0aa81f223a21-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-53e83-predictor-78879675f8-nsgwk\" (UID: \"b4e94490-b45d-4292-baf1-0aa81f223a21\") " pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:12:57.273626 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.273534    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4e94490-b45d-4292-baf1-0aa81f223a21-proxy-tls\") pod \"message-dumper-raw-53e83-predictor-78879675f8-nsgwk\" (UID: \"b4e94490-b45d-4292-baf1-0aa81f223a21\") " pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:12:57.273626 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.273557    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vt4n\" (UniqueName: \"kubernetes.io/projected/b4e94490-b45d-4292-baf1-0aa81f223a21-kube-api-access-8vt4n\") pod \"message-dumper-raw-53e83-predictor-78879675f8-nsgwk\" (UID: \"b4e94490-b45d-4292-baf1-0aa81f223a21\") " pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:12:57.274234 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.274213    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4e94490-b45d-4292-baf1-0aa81f223a21-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-53e83-predictor-78879675f8-nsgwk\" (UID: \"b4e94490-b45d-4292-baf1-0aa81f223a21\") " pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:12:57.275895 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.275877    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4e94490-b45d-4292-baf1-0aa81f223a21-proxy-tls\") pod \"message-dumper-raw-53e83-predictor-78879675f8-nsgwk\" (UID: \"b4e94490-b45d-4292-baf1-0aa81f223a21\") " pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:12:57.281311 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.281291    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vt4n\" (UniqueName: \"kubernetes.io/projected/b4e94490-b45d-4292-baf1-0aa81f223a21-kube-api-access-8vt4n\") pod \"message-dumper-raw-53e83-predictor-78879675f8-nsgwk\" (UID: \"b4e94490-b45d-4292-baf1-0aa81f223a21\") " pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:12:57.371708 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.371681    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:12:57.500881 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.500858    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"]
Jul 08 07:12:57.502535 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:12:57.502501    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e94490_b45d_4292_baf1_0aa81f223a21.slice/crio-a4eac4a7f64cc95146352afe37beeb3f897d82be1fe271ca00924f59464f3dcc WatchSource:0}: Error finding container a4eac4a7f64cc95146352afe37beeb3f897d82be1fe271ca00924f59464f3dcc: Status 404 returned error can't find the container with id a4eac4a7f64cc95146352afe37beeb3f897d82be1fe271ca00924f59464f3dcc
Jul 08 07:12:57.504319 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.504304    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 07:12:57.992951 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.992922    2557 generic.go:356] "Generic (PLEG): container finished" podID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerID="5d1177799579431046c28cbdac6b89da10d9ce31d1bce7300645ba16c20faea9" exitCode=2
Jul 08 07:12:57.993125 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.992994    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" event={"ID":"59ade4e3-fb68-4453-95ec-6f8ac3f136a8","Type":"ContainerDied","Data":"5d1177799579431046c28cbdac6b89da10d9ce31d1bce7300645ba16c20faea9"}
Jul 08 07:12:57.994162 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.994137    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk" event={"ID":"b4e94490-b45d-4292-baf1-0aa81f223a21","Type":"ContainerStarted","Data":"a4eac4a7f64cc95146352afe37beeb3f897d82be1fe271ca00924f59464f3dcc"}
Jul 08 07:12:57.996056 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.996036    2557 generic.go:356] "Generic (PLEG): container finished" podID="a93541e4-a124-469f-903b-c22754a2481d" containerID="80f3f8957725440faef02fefcddd47104e6021d5cace233a88072f5832df07ee" exitCode=2
Jul 08 07:12:57.996128 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:57.996072    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" event={"ID":"a93541e4-a124-469f-903b-c22754a2481d","Type":"ContainerDied","Data":"80f3f8957725440faef02fefcddd47104e6021d5cace233a88072f5832df07ee"}
Jul 08 07:12:58.569577 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:58.569529    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 08 07:12:59.001700 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:59.001668    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk" event={"ID":"b4e94490-b45d-4292-baf1-0aa81f223a21","Type":"ContainerStarted","Data":"3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072"}
Jul 08 07:12:59.001896 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:59.001733    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk" event={"ID":"b4e94490-b45d-4292-baf1-0aa81f223a21","Type":"ContainerStarted","Data":"04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8"}
Jul 08 07:12:59.001896 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:59.001863    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:12:59.020604 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:59.020562    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk" podStartSLOduration=0.732642728 podStartE2EDuration="2.020547789s" totalImagesPullingTime="1.287905061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:12:59.019138918 +0000 UTC m=+1111.755483632" watchObservedRunningTime="2026-07-08 07:12:59.020547789 +0000 UTC m=+1111.756892502"
Jul 08 07:12:59.575118 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:59.575071    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 08 07:12:59.579411 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:59.579386    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 08 07:12:59.781393 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:12:59.781350    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 08 07:13:00.005356 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:00.005322    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:13:00.006900 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:00.006883    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:13:00.835404 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:00.835379    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:13:00.904193 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:00.904164    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-cabundle-cert\") pod \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") "
Jul 08 07:13:00.904349 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:00.904218    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-proxy-tls\") pod \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") "
Jul 08 07:13:00.904349 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:00.904256    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kserve-provision-location\") pod \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") "
Jul 08 07:13:00.904349 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:00.904272    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swpt2\" (UniqueName: \"kubernetes.io/projected/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kube-api-access-swpt2\") pod \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") "
Jul 08 07:13:00.904349 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:00.904290    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kube-rbac-proxy-sar-config\") pod \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\" (UID: \"59ade4e3-fb68-4453-95ec-6f8ac3f136a8\") "
Jul 08 07:13:00.904589 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:00.904557    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "59ade4e3-fb68-4453-95ec-6f8ac3f136a8" (UID: "59ade4e3-fb68-4453-95ec-6f8ac3f136a8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:13:00.904675 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:00.904650    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59ade4e3-fb68-4453-95ec-6f8ac3f136a8" (UID: "59ade4e3-fb68-4453-95ec-6f8ac3f136a8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:13:00.904742 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:00.904722    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "59ade4e3-fb68-4453-95ec-6f8ac3f136a8" (UID: "59ade4e3-fb68-4453-95ec-6f8ac3f136a8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:13:00.906438 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:00.906415    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kube-api-access-swpt2" (OuterVolumeSpecName: "kube-api-access-swpt2") pod "59ade4e3-fb68-4453-95ec-6f8ac3f136a8" (UID: "59ade4e3-fb68-4453-95ec-6f8ac3f136a8"). InnerVolumeSpecName "kube-api-access-swpt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:13:00.906438 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:00.906420    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "59ade4e3-fb68-4453-95ec-6f8ac3f136a8" (UID: "59ade4e3-fb68-4453-95ec-6f8ac3f136a8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:13:01.005729 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.005700    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-cabundle-cert\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:13:01.005729 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.005726    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-proxy-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:13:01.005896 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.005739    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kserve-provision-location\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:13:01.005896 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.005751    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swpt2\" (UniqueName: \"kubernetes.io/projected/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kube-api-access-swpt2\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:13:01.005896 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.005783    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ade4e3-fb68-4453-95ec-6f8ac3f136a8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:13:01.010488 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.010461    2557 generic.go:356] "Generic (PLEG): container finished" podID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerID="ba891c312923545aa11604c66185196e8be0962f11bb74992375e6b301407f0f" exitCode=0
Jul 08 07:13:01.010585 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.010511    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"
Jul 08 07:13:01.010585 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.010552    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" event={"ID":"59ade4e3-fb68-4453-95ec-6f8ac3f136a8","Type":"ContainerDied","Data":"ba891c312923545aa11604c66185196e8be0962f11bb74992375e6b301407f0f"}
Jul 08 07:13:01.010702 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.010622    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v" event={"ID":"59ade4e3-fb68-4453-95ec-6f8ac3f136a8","Type":"ContainerDied","Data":"f428f4f81cab583c2e3ff3a2799b2b01c04739e8bf7ab99384f02cab821bb7cb"}
Jul 08 07:13:01.010702 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.010664    2557 scope.go:117] "RemoveContainer" containerID="5d1177799579431046c28cbdac6b89da10d9ce31d1bce7300645ba16c20faea9"
Jul 08 07:13:01.018789 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.018749    2557 scope.go:117] "RemoveContainer" containerID="ba891c312923545aa11604c66185196e8be0962f11bb74992375e6b301407f0f"
Jul 08 07:13:01.026082 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.026067    2557 scope.go:117] "RemoveContainer" containerID="0a35864568dc60b939ebbba491d12cdecfcee68e34df6dc9f47870914f6e048f"
Jul 08 07:13:01.031121 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.031100    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"]
Jul 08 07:13:01.033506 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.033493    2557 scope.go:117] "RemoveContainer" containerID="5d1177799579431046c28cbdac6b89da10d9ce31d1bce7300645ba16c20faea9"
Jul 08 07:13:01.033749 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:13:01.033732    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1177799579431046c28cbdac6b89da10d9ce31d1bce7300645ba16c20faea9\": container with ID starting with 5d1177799579431046c28cbdac6b89da10d9ce31d1bce7300645ba16c20faea9 not found: ID does not exist" containerID="5d1177799579431046c28cbdac6b89da10d9ce31d1bce7300645ba16c20faea9"
Jul 08 07:13:01.033820 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.033777    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1177799579431046c28cbdac6b89da10d9ce31d1bce7300645ba16c20faea9"} err="failed to get container status \"5d1177799579431046c28cbdac6b89da10d9ce31d1bce7300645ba16c20faea9\": rpc error: code = NotFound desc = could not find container \"5d1177799579431046c28cbdac6b89da10d9ce31d1bce7300645ba16c20faea9\": container with ID starting with 5d1177799579431046c28cbdac6b89da10d9ce31d1bce7300645ba16c20faea9 not found: ID does not exist"
Jul 08 07:13:01.033820 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.033793    2557 scope.go:117] "RemoveContainer" containerID="ba891c312923545aa11604c66185196e8be0962f11bb74992375e6b301407f0f"
Jul 08 07:13:01.034043 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:13:01.034016    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba891c312923545aa11604c66185196e8be0962f11bb74992375e6b301407f0f\": container with ID starting with ba891c312923545aa11604c66185196e8be0962f11bb74992375e6b301407f0f not found: ID does not exist" containerID="ba891c312923545aa11604c66185196e8be0962f11bb74992375e6b301407f0f"
Jul 08 07:13:01.034139 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.034048    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba891c312923545aa11604c66185196e8be0962f11bb74992375e6b301407f0f"} err="failed to get container status \"ba891c312923545aa11604c66185196e8be0962f11bb74992375e6b301407f0f\": rpc error: code = NotFound desc = could not find container \"ba891c312923545aa11604c66185196e8be0962f11bb74992375e6b301407f0f\": container with ID starting with ba891c312923545aa11604c66185196e8be0962f11bb74992375e6b301407f0f not found: ID does not exist"
Jul 08 07:13:01.034139 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.034063    2557 scope.go:117] "RemoveContainer" containerID="0a35864568dc60b939ebbba491d12cdecfcee68e34df6dc9f47870914f6e048f"
Jul 08 07:13:01.034472 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:13:01.034438    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a35864568dc60b939ebbba491d12cdecfcee68e34df6dc9f47870914f6e048f\": container with ID starting with 0a35864568dc60b939ebbba491d12cdecfcee68e34df6dc9f47870914f6e048f not found: ID does not exist" containerID="0a35864568dc60b939ebbba491d12cdecfcee68e34df6dc9f47870914f6e048f"
Jul 08 07:13:01.034583 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.034473    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a35864568dc60b939ebbba491d12cdecfcee68e34df6dc9f47870914f6e048f"} err="failed to get container status \"0a35864568dc60b939ebbba491d12cdecfcee68e34df6dc9f47870914f6e048f\": rpc error: code = NotFound desc = could not find container \"0a35864568dc60b939ebbba491d12cdecfcee68e34df6dc9f47870914f6e048f\": container with ID starting with 0a35864568dc60b939ebbba491d12cdecfcee68e34df6dc9f47870914f6e048f not found: ID does not exist"
Jul 08 07:13:01.036334 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.036314    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-aea5a-predictor-695b4769df-w926v"]
Jul 08 07:13:01.672205 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.672181    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:13:01.711796 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.711749    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a93541e4-a124-469f-903b-c22754a2481d-proxy-tls\") pod \"a93541e4-a124-469f-903b-c22754a2481d\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") "
Jul 08 07:13:01.711906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.711830    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a93541e4-a124-469f-903b-c22754a2481d-cabundle-cert\") pod \"a93541e4-a124-469f-903b-c22754a2481d\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") "
Jul 08 07:13:01.711906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.711848    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a93541e4-a124-469f-903b-c22754a2481d-kserve-provision-location\") pod \"a93541e4-a124-469f-903b-c22754a2481d\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") "
Jul 08 07:13:01.711906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.711899    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a93541e4-a124-469f-903b-c22754a2481d-kube-rbac-proxy-sar-config\") pod \"a93541e4-a124-469f-903b-c22754a2481d\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") "
Jul 08 07:13:01.712008 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.711957    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxqtr\" (UniqueName: \"kubernetes.io/projected/a93541e4-a124-469f-903b-c22754a2481d-kube-api-access-jxqtr\") pod \"a93541e4-a124-469f-903b-c22754a2481d\" (UID: \"a93541e4-a124-469f-903b-c22754a2481d\") "
Jul 08 07:13:01.712242 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.712216    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93541e4-a124-469f-903b-c22754a2481d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a93541e4-a124-469f-903b-c22754a2481d" (UID: "a93541e4-a124-469f-903b-c22754a2481d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:13:01.712242 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.712217    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93541e4-a124-469f-903b-c22754a2481d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a93541e4-a124-469f-903b-c22754a2481d" (UID: "a93541e4-a124-469f-903b-c22754a2481d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:13:01.712372 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.712320    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93541e4-a124-469f-903b-c22754a2481d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a93541e4-a124-469f-903b-c22754a2481d" (UID: "a93541e4-a124-469f-903b-c22754a2481d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:13:01.713900 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.713881    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a93541e4-a124-469f-903b-c22754a2481d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a93541e4-a124-469f-903b-c22754a2481d" (UID: "a93541e4-a124-469f-903b-c22754a2481d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:13:01.714138 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.714108    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93541e4-a124-469f-903b-c22754a2481d-kube-api-access-jxqtr" (OuterVolumeSpecName: "kube-api-access-jxqtr") pod "a93541e4-a124-469f-903b-c22754a2481d" (UID: "a93541e4-a124-469f-903b-c22754a2481d"). InnerVolumeSpecName "kube-api-access-jxqtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:13:01.785341 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.785282    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" path="/var/lib/kubelet/pods/59ade4e3-fb68-4453-95ec-6f8ac3f136a8/volumes"
Jul 08 07:13:01.813310 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.813292    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a93541e4-a124-469f-903b-c22754a2481d-cabundle-cert\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:13:01.813398 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.813313    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a93541e4-a124-469f-903b-c22754a2481d-kserve-provision-location\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:13:01.813398 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.813325    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a93541e4-a124-469f-903b-c22754a2481d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:13:01.813398 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.813334    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxqtr\" (UniqueName: \"kubernetes.io/projected/a93541e4-a124-469f-903b-c22754a2481d-kube-api-access-jxqtr\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:13:01.813398 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:01.813343    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a93541e4-a124-469f-903b-c22754a2481d-proxy-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:13:02.017464 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.017432    2557 generic.go:356] "Generic (PLEG): container finished" podID="a93541e4-a124-469f-903b-c22754a2481d" containerID="f10deb6ddc15b9b10b518d62433618f58d3730a35375f1af11e673de71da0d88" exitCode=0
Jul 08 07:13:02.017893 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.017484    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"
Jul 08 07:13:02.017893 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.017490    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" event={"ID":"a93541e4-a124-469f-903b-c22754a2481d","Type":"ContainerDied","Data":"f10deb6ddc15b9b10b518d62433618f58d3730a35375f1af11e673de71da0d88"}
Jul 08 07:13:02.017893 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.017540    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb" event={"ID":"a93541e4-a124-469f-903b-c22754a2481d","Type":"ContainerDied","Data":"96c20700ea23c52bd85e144bd6d74a8cadee2a536aafc2aadc15ee472c96c075"}
Jul 08 07:13:02.017893 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.017566    2557 scope.go:117] "RemoveContainer" containerID="80f3f8957725440faef02fefcddd47104e6021d5cace233a88072f5832df07ee"
Jul 08 07:13:02.026083 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.026067    2557 scope.go:117] "RemoveContainer" containerID="f10deb6ddc15b9b10b518d62433618f58d3730a35375f1af11e673de71da0d88"
Jul 08 07:13:02.033454 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.033434    2557 scope.go:117] "RemoveContainer" containerID="28f81015707d961255f21eb54923dcc1894c60b34ab19c66c0a0fe4613601840"
Jul 08 07:13:02.033613 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.033597    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"]
Jul 08 07:13:02.039621 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.039570    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-aea5a-predictor-7b7677b79-mjzwb"]
Jul 08 07:13:02.041278 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.041262    2557 scope.go:117] "RemoveContainer" containerID="80f3f8957725440faef02fefcddd47104e6021d5cace233a88072f5832df07ee"
Jul 08 07:13:02.041528 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:13:02.041509    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f3f8957725440faef02fefcddd47104e6021d5cace233a88072f5832df07ee\": container with ID starting with 80f3f8957725440faef02fefcddd47104e6021d5cace233a88072f5832df07ee not found: ID does not exist" containerID="80f3f8957725440faef02fefcddd47104e6021d5cace233a88072f5832df07ee"
Jul 08 07:13:02.041570 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.041536    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f3f8957725440faef02fefcddd47104e6021d5cace233a88072f5832df07ee"} err="failed to get container status \"80f3f8957725440faef02fefcddd47104e6021d5cace233a88072f5832df07ee\": rpc error: code = NotFound desc = could not find container \"80f3f8957725440faef02fefcddd47104e6021d5cace233a88072f5832df07ee\": container with ID starting with 80f3f8957725440faef02fefcddd47104e6021d5cace233a88072f5832df07ee not found: ID does not exist"
Jul 08 07:13:02.041570 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.041552    2557 scope.go:117] "RemoveContainer" containerID="f10deb6ddc15b9b10b518d62433618f58d3730a35375f1af11e673de71da0d88"
Jul 08 07:13:02.041826 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:13:02.041808    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10deb6ddc15b9b10b518d62433618f58d3730a35375f1af11e673de71da0d88\": container with ID starting with f10deb6ddc15b9b10b518d62433618f58d3730a35375f1af11e673de71da0d88 not found: ID does not exist" containerID="f10deb6ddc15b9b10b518d62433618f58d3730a35375f1af11e673de71da0d88"
Jul 08 07:13:02.041876 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.041832    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10deb6ddc15b9b10b518d62433618f58d3730a35375f1af11e673de71da0d88"} err="failed to get container status \"f10deb6ddc15b9b10b518d62433618f58d3730a35375f1af11e673de71da0d88\": rpc error: code = NotFound desc = could not find container \"f10deb6ddc15b9b10b518d62433618f58d3730a35375f1af11e673de71da0d88\": container with ID starting with f10deb6ddc15b9b10b518d62433618f58d3730a35375f1af11e673de71da0d88 not found: ID does not exist"
Jul 08 07:13:02.041876 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.041847    2557 scope.go:117] "RemoveContainer" containerID="28f81015707d961255f21eb54923dcc1894c60b34ab19c66c0a0fe4613601840"
Jul 08 07:13:02.042064 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:13:02.042048    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f81015707d961255f21eb54923dcc1894c60b34ab19c66c0a0fe4613601840\": container with ID starting with 28f81015707d961255f21eb54923dcc1894c60b34ab19c66c0a0fe4613601840 not found: ID does not exist" containerID="28f81015707d961255f21eb54923dcc1894c60b34ab19c66c0a0fe4613601840"
Jul 08 07:13:02.042106 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:02.042068    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f81015707d961255f21eb54923dcc1894c60b34ab19c66c0a0fe4613601840"} err="failed to get container status \"28f81015707d961255f21eb54923dcc1894c60b34ab19c66c0a0fe4613601840\": rpc error: code = NotFound desc = could not find container \"28f81015707d961255f21eb54923dcc1894c60b34ab19c66c0a0fe4613601840\": container with ID starting with 28f81015707d961255f21eb54923dcc1894c60b34ab19c66c0a0fe4613601840 not found: ID does not exist"
Jul 08 07:13:03.785604 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:03.785570    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a93541e4-a124-469f-903b-c22754a2481d" path="/var/lib/kubelet/pods/a93541e4-a124-469f-903b-c22754a2481d/volumes"
Jul 08 07:13:07.022333 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:07.022308    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:13:17.109950 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.109916    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"]
Jul 08 07:13:17.110330 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110258    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="storage-initializer"
Jul 08 07:13:17.110330 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110270    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="storage-initializer"
Jul 08 07:13:17.110330 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110279    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="storage-initializer"
Jul 08 07:13:17.110330 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110285    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="storage-initializer"
Jul 08 07:13:17.110330 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110291    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kserve-container"
Jul 08 07:13:17.110330 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110296    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kserve-container"
Jul 08 07:13:17.110330 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110302    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kube-rbac-proxy"
Jul 08 07:13:17.110330 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110307    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kube-rbac-proxy"
Jul 08 07:13:17.110330 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110314    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kube-rbac-proxy"
Jul 08 07:13:17.110330 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110319    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kube-rbac-proxy"
Jul 08 07:13:17.110330 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110328    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container"
Jul 08 07:13:17.110330 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110333    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container"
Jul 08 07:13:17.110788 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110385    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kserve-container"
Jul 08 07:13:17.110788 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110394    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93541e4-a124-469f-903b-c22754a2481d" containerName="kube-rbac-proxy"
Jul 08 07:13:17.110788 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110399    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kube-rbac-proxy"
Jul 08 07:13:17.110788 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.110407    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="59ade4e3-fb68-4453-95ec-6f8ac3f136a8" containerName="kserve-container"
Jul 08 07:13:17.115238 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.115219    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.117772 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.117740    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-53e83-predictor-serving-cert\""
Jul 08 07:13:17.117897 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.117773    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-53e83-kube-rbac-proxy-sar-config\""
Jul 08 07:13:17.117897 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.117780    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 07:13:17.117897 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.117796    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 07:13:17.124939 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.124914    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"]
Jul 08 07:13:17.234313 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.234285    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-cabundle-cert\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.234435 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.234319    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-proxy-tls\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.234435 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.234343    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kserve-provision-location\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.234435 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.234408    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.234553 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.234474    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vfzd\" (UniqueName: \"kubernetes.io/projected/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kube-api-access-8vfzd\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.335818 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.335782    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vfzd\" (UniqueName: \"kubernetes.io/projected/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kube-api-access-8vfzd\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.335974 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.335859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-cabundle-cert\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.335974 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.335888    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-proxy-tls\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.335974 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.335921    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kserve-provision-location\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.335974 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.335947    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.336193 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:13:17.335999    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-serving-cert: secret "isvc-logger-raw-53e83-predictor-serving-cert" not found
Jul 08 07:13:17.336193 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:13:17.336072    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-proxy-tls podName:3dce2faa-25c2-4a73-9fc2-75885d71fd8e nodeName:}" failed. No retries permitted until 2026-07-08 07:13:17.836052292 +0000 UTC m=+1130.572396998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-proxy-tls") pod "isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" (UID: "3dce2faa-25c2-4a73-9fc2-75885d71fd8e") : secret "isvc-logger-raw-53e83-predictor-serving-cert" not found
Jul 08 07:13:17.336394 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.336373    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kserve-provision-location\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.336601 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.336585    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-cabundle-cert\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.336659 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.336638    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.346800 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.346738    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vfzd\" (UniqueName: \"kubernetes.io/projected/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kube-api-access-8vfzd\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.839955 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.839920    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-proxy-tls\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:17.842551 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:17.842517    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-proxy-tls\") pod \"isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") " pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:18.026737 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:18.026716    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:18.159895 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:18.159871    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"]
Jul 08 07:13:18.161794 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:13:18.161747    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dce2faa_25c2_4a73_9fc2_75885d71fd8e.slice/crio-4a81d1fbbf611d336c81dfac07fa9e5a8d8863043b0502977a80f53de44966a1 WatchSource:0}: Error finding container 4a81d1fbbf611d336c81dfac07fa9e5a8d8863043b0502977a80f53de44966a1: Status 404 returned error can't find the container with id 4a81d1fbbf611d336c81dfac07fa9e5a8d8863043b0502977a80f53de44966a1
Jul 08 07:13:19.090094 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:19.090060    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" event={"ID":"3dce2faa-25c2-4a73-9fc2-75885d71fd8e","Type":"ContainerStarted","Data":"7dc159f698b4d083f315679ca7612d750b669de26d7029d5c813a020940ac12d"}
Jul 08 07:13:19.090265 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:19.090133    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" event={"ID":"3dce2faa-25c2-4a73-9fc2-75885d71fd8e","Type":"ContainerStarted","Data":"4a81d1fbbf611d336c81dfac07fa9e5a8d8863043b0502977a80f53de44966a1"}
Jul 08 07:13:22.104961 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:22.104868    2557 generic.go:356] "Generic (PLEG): container finished" podID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerID="7dc159f698b4d083f315679ca7612d750b669de26d7029d5c813a020940ac12d" exitCode=0
Jul 08 07:13:22.104961 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:22.104944    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" event={"ID":"3dce2faa-25c2-4a73-9fc2-75885d71fd8e","Type":"ContainerDied","Data":"7dc159f698b4d083f315679ca7612d750b669de26d7029d5c813a020940ac12d"}
Jul 08 07:13:23.111358 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:23.111326    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" event={"ID":"3dce2faa-25c2-4a73-9fc2-75885d71fd8e","Type":"ContainerStarted","Data":"1dd38d765c3a0dda6d6eec460cbdedb4ccb6699163d7f59e51bc9bce05ca3aa0"}
Jul 08 07:13:23.111705 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:23.111398    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" event={"ID":"3dce2faa-25c2-4a73-9fc2-75885d71fd8e","Type":"ContainerStarted","Data":"c71089ec4b0cbc3d67a2e7f646dc65c26402dfa2cd611630cee43a257a6f55ca"}
Jul 08 07:13:23.111705 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:23.111423    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" event={"ID":"3dce2faa-25c2-4a73-9fc2-75885d71fd8e","Type":"ContainerStarted","Data":"83793913ee055a274ed64ea9b0a717ab53dd8e0d75f85bb929ef6e28c7892eb6"}
Jul 08 07:13:23.111850 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:23.111709    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:23.111900 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:23.111887    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:23.113157 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:23.113125    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 07:13:23.135002 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:23.134962    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podStartSLOduration=2.574222867 podStartE2EDuration="6.134948095s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.560725228s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:13:23.134082194 +0000 UTC m=+1135.870426907" watchObservedRunningTime="2026-07-08 07:13:23.134948095 +0000 UTC m=+1135.871292857"
Jul 08 07:13:24.116255 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:24.116216    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 07:13:24.116830 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:24.116278    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:24.117536 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:24.117498    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:13:25.121123 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:25.121081    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 07:13:25.121599 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:25.121577    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:13:30.125717 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:30.125686    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:13:30.126322 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:30.126282    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 07:13:30.126585 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:30.126560    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:13:40.126316 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:40.126264    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 07:13:40.126822 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:40.126791    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:13:50.126870 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:50.126739    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 07:13:50.127314 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:13:50.127276    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:14:00.127187 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:00.127147    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 07:14:00.127980 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:00.127696    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:14:10.126243 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:10.126206    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 07:14:10.126788 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:10.126736    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:14:20.127170 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:20.127130    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 07:14:20.127710 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:20.127624    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:14:30.126683 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:30.126640    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 07:14:30.127199 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:30.127149    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:14:40.126932 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:40.126900    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:14:40.127500 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:40.127443    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:14:52.136748 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.136715    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-53e83-predictor-78879675f8-nsgwk_b4e94490-b45d-4292-baf1-0aa81f223a21/kserve-container/0.log"
Jul 08 07:14:52.390977 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.390898    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"]
Jul 08 07:14:52.391198 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.391153    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container" containerID="cri-o://83793913ee055a274ed64ea9b0a717ab53dd8e0d75f85bb929ef6e28c7892eb6" gracePeriod=30
Jul 08 07:14:52.391198 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.391177    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent" containerID="cri-o://1dd38d765c3a0dda6d6eec460cbdedb4ccb6699163d7f59e51bc9bce05ca3aa0" gracePeriod=30
Jul 08 07:14:52.391350 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.391199    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kube-rbac-proxy" containerID="cri-o://c71089ec4b0cbc3d67a2e7f646dc65c26402dfa2cd611630cee43a257a6f55ca" gracePeriod=30
Jul 08 07:14:52.494182 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.494155    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"]
Jul 08 07:14:52.499394 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.499370    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.501684 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.501661    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-bba5b-predictor-serving-cert\""
Jul 08 07:14:52.502020 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.502001    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-bba5b-kube-rbac-proxy-sar-config\""
Jul 08 07:14:52.509074 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.509052    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"]
Jul 08 07:14:52.525193 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.525173    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"]
Jul 08 07:14:52.525372 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.525352    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk" podUID="b4e94490-b45d-4292-baf1-0aa81f223a21" containerName="kserve-container" containerID="cri-o://04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8" gracePeriod=30
Jul 08 07:14:52.525448 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.525428    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk" podUID="b4e94490-b45d-4292-baf1-0aa81f223a21" containerName="kube-rbac-proxy" containerID="cri-o://3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072" gracePeriod=30
Jul 08 07:14:52.586052 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.586025    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-proxy-tls\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.586156 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.586072    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-cabundle-cert\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.586200 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.586139    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6pd7\" (UniqueName: \"kubernetes.io/projected/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kube-api-access-m6pd7\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.586200 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.586179    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.586271 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.586199    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.687085 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.687053    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-proxy-tls\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.687246 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.687102    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-cabundle-cert\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.687246 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.687143    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6pd7\" (UniqueName: \"kubernetes.io/projected/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kube-api-access-m6pd7\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.687246 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.687162    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.687246 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.687183    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.687246 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:14:52.687222    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-serving-cert: secret "isvc-sklearn-scale-raw-bba5b-predictor-serving-cert" not found
Jul 08 07:14:52.687508 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:14:52.687294    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-proxy-tls podName:395bc3c1-2bdc-4dbd-aa3f-82ef64696d25 nodeName:}" failed. No retries permitted until 2026-07-08 07:14:53.187272381 +0000 UTC m=+1225.923617095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-proxy-tls") pod "isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" (UID: "395bc3c1-2bdc-4dbd-aa3f-82ef64696d25") : secret "isvc-sklearn-scale-raw-bba5b-predictor-serving-cert" not found
Jul 08 07:14:52.687658 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.687636    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.687921 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.687898    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-cabundle-cert\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.687921 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.687914    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.696120 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.696097    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6pd7\" (UniqueName: \"kubernetes.io/projected/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kube-api-access-m6pd7\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:52.792034 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.792009    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:14:52.888525 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.888501    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4e94490-b45d-4292-baf1-0aa81f223a21-proxy-tls\") pod \"b4e94490-b45d-4292-baf1-0aa81f223a21\" (UID: \"b4e94490-b45d-4292-baf1-0aa81f223a21\") "
Jul 08 07:14:52.888668 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.888550    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4e94490-b45d-4292-baf1-0aa81f223a21-kube-rbac-proxy-sar-config\") pod \"b4e94490-b45d-4292-baf1-0aa81f223a21\" (UID: \"b4e94490-b45d-4292-baf1-0aa81f223a21\") "
Jul 08 07:14:52.888668 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.888586    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vt4n\" (UniqueName: \"kubernetes.io/projected/b4e94490-b45d-4292-baf1-0aa81f223a21-kube-api-access-8vt4n\") pod \"b4e94490-b45d-4292-baf1-0aa81f223a21\" (UID: \"b4e94490-b45d-4292-baf1-0aa81f223a21\") "
Jul 08 07:14:52.888971 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.888941    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e94490-b45d-4292-baf1-0aa81f223a21-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b4e94490-b45d-4292-baf1-0aa81f223a21" (UID: "b4e94490-b45d-4292-baf1-0aa81f223a21"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:14:52.890607 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.890580    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e94490-b45d-4292-baf1-0aa81f223a21-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4e94490-b45d-4292-baf1-0aa81f223a21" (UID: "b4e94490-b45d-4292-baf1-0aa81f223a21"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:14:52.890696 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.890667    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e94490-b45d-4292-baf1-0aa81f223a21-kube-api-access-8vt4n" (OuterVolumeSpecName: "kube-api-access-8vt4n") pod "b4e94490-b45d-4292-baf1-0aa81f223a21" (UID: "b4e94490-b45d-4292-baf1-0aa81f223a21"). InnerVolumeSpecName "kube-api-access-8vt4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:14:52.989570 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.989548    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4e94490-b45d-4292-baf1-0aa81f223a21-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:14:52.989570 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.989568    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vt4n\" (UniqueName: \"kubernetes.io/projected/b4e94490-b45d-4292-baf1-0aa81f223a21-kube-api-access-8vt4n\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:14:52.989720 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:52.989579    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4e94490-b45d-4292-baf1-0aa81f223a21-proxy-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:14:53.191692 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.191663    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-proxy-tls\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:53.194049 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.194033    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-proxy-tls\") pod \"isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:53.412657 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.412622    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:53.500060 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.500024    2557 generic.go:356] "Generic (PLEG): container finished" podID="b4e94490-b45d-4292-baf1-0aa81f223a21" containerID="3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072" exitCode=2
Jul 08 07:14:53.500060 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.500059    2557 generic.go:356] "Generic (PLEG): container finished" podID="b4e94490-b45d-4292-baf1-0aa81f223a21" containerID="04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8" exitCode=2
Jul 08 07:14:53.500289 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.500079    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"
Jul 08 07:14:53.500289 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.500103    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk" event={"ID":"b4e94490-b45d-4292-baf1-0aa81f223a21","Type":"ContainerDied","Data":"3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072"}
Jul 08 07:14:53.500289 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.500175    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk" event={"ID":"b4e94490-b45d-4292-baf1-0aa81f223a21","Type":"ContainerDied","Data":"04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8"}
Jul 08 07:14:53.500289 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.500203    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk" event={"ID":"b4e94490-b45d-4292-baf1-0aa81f223a21","Type":"ContainerDied","Data":"a4eac4a7f64cc95146352afe37beeb3f897d82be1fe271ca00924f59464f3dcc"}
Jul 08 07:14:53.500289 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.500224    2557 scope.go:117] "RemoveContainer" containerID="3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072"
Jul 08 07:14:53.503305 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.503285    2557 generic.go:356] "Generic (PLEG): container finished" podID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerID="c71089ec4b0cbc3d67a2e7f646dc65c26402dfa2cd611630cee43a257a6f55ca" exitCode=2
Jul 08 07:14:53.503423 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.503338    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" event={"ID":"3dce2faa-25c2-4a73-9fc2-75885d71fd8e","Type":"ContainerDied","Data":"c71089ec4b0cbc3d67a2e7f646dc65c26402dfa2cd611630cee43a257a6f55ca"}
Jul 08 07:14:53.510792 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.510775    2557 scope.go:117] "RemoveContainer" containerID="04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8"
Jul 08 07:14:53.520347 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.520323    2557 scope.go:117] "RemoveContainer" containerID="3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072"
Jul 08 07:14:53.521259 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:14:53.521230    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072\": container with ID starting with 3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072 not found: ID does not exist" containerID="3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072"
Jul 08 07:14:53.521356 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.521271    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072"} err="failed to get container status \"3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072\": rpc error: code = NotFound desc = could not find container \"3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072\": container with ID starting with 3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072 not found: ID does not exist"
Jul 08 07:14:53.521356 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.521289    2557 scope.go:117] "RemoveContainer" containerID="04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8"
Jul 08 07:14:53.521556 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:14:53.521534    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8\": container with ID starting with 04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8 not found: ID does not exist" containerID="04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8"
Jul 08 07:14:53.521612 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.521563    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8"} err="failed to get container status \"04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8\": rpc error: code = NotFound desc = could not find container \"04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8\": container with ID starting with 04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8 not found: ID does not exist"
Jul 08 07:14:53.521612 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.521582    2557 scope.go:117] "RemoveContainer" containerID="3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072"
Jul 08 07:14:53.522051 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.522030    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072"} err="failed to get container status \"3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072\": rpc error: code = NotFound desc = could not find container \"3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072\": container with ID starting with 3651ec7bbed5cb54f8b66f6016a8d5adfdef1cba887b463a440c90a721db5072 not found: ID does not exist"
Jul 08 07:14:53.522148 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.522052    2557 scope.go:117] "RemoveContainer" containerID="04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8"
Jul 08 07:14:53.522347 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.522315    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8"} err="failed to get container status \"04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8\": rpc error: code = NotFound desc = could not find container \"04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8\": container with ID starting with 04fe1378a871c7d8f37f087e477d2e05285bce0909884f93fe132e5d3a9bc0c8 not found: ID does not exist"
Jul 08 07:14:53.523437 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.523413    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"]
Jul 08 07:14:53.525653 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.525633    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-53e83-predictor-78879675f8-nsgwk"]
Jul 08 07:14:53.550085 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.550042    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"]
Jul 08 07:14:53.552452 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:14:53.552434    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395bc3c1_2bdc_4dbd_aa3f_82ef64696d25.slice/crio-247aace483389ee3cc408f44970b1146e5d2cbfc660f897bdbc6e57df3bcf449 WatchSource:0}: Error finding container 247aace483389ee3cc408f44970b1146e5d2cbfc660f897bdbc6e57df3bcf449: Status 404 returned error can't find the container with id 247aace483389ee3cc408f44970b1146e5d2cbfc660f897bdbc6e57df3bcf449
Jul 08 07:14:53.792405 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:53.792364    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4e94490-b45d-4292-baf1-0aa81f223a21" path="/var/lib/kubelet/pods/b4e94490-b45d-4292-baf1-0aa81f223a21/volumes"
Jul 08 07:14:54.509234 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:54.509196    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" event={"ID":"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25","Type":"ContainerStarted","Data":"5c18ac7166e1b3ae03a599276d8cd187e33801d4d23632643855ff51ade264d5"}
Jul 08 07:14:54.509681 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:54.509261    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" event={"ID":"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25","Type":"ContainerStarted","Data":"247aace483389ee3cc408f44970b1146e5d2cbfc660f897bdbc6e57df3bcf449"}
Jul 08 07:14:55.121323 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:55.121286    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 08 07:14:56.521341 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:56.521310    2557 generic.go:356] "Generic (PLEG): container finished" podID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerID="83793913ee055a274ed64ea9b0a717ab53dd8e0d75f85bb929ef6e28c7892eb6" exitCode=0
Jul 08 07:14:56.521887 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:56.521359    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" event={"ID":"3dce2faa-25c2-4a73-9fc2-75885d71fd8e","Type":"ContainerDied","Data":"83793913ee055a274ed64ea9b0a717ab53dd8e0d75f85bb929ef6e28c7892eb6"}
Jul 08 07:14:57.527257 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:57.527228    2557 generic.go:356] "Generic (PLEG): container finished" podID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerID="5c18ac7166e1b3ae03a599276d8cd187e33801d4d23632643855ff51ade264d5" exitCode=0
Jul 08 07:14:57.527588 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:57.527309    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" event={"ID":"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25","Type":"ContainerDied","Data":"5c18ac7166e1b3ae03a599276d8cd187e33801d4d23632643855ff51ade264d5"}
Jul 08 07:14:58.533831 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:58.533796    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" event={"ID":"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25","Type":"ContainerStarted","Data":"f5690e1a0841de90a2b75151e93d7982586d263359395530b6e1062878e31157"}
Jul 08 07:14:58.534277 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:58.533858    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" event={"ID":"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25","Type":"ContainerStarted","Data":"95b51fa31ca8fd84d7db4e059d65146b96bf415622bdc5ab87942b2f2265aa81"}
Jul 08 07:14:58.534277 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:58.534176    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:58.534277 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:58.534220    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:14:58.535408 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:58.535381    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:14:58.553880 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:58.553840    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podStartSLOduration=2.741336139 podStartE2EDuration="6.553822044s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.812485905s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:14:58.551662441 +0000 UTC m=+1231.288007156" watchObservedRunningTime="2026-07-08 07:14:58.553822044 +0000 UTC m=+1231.290166758"
Jul 08 07:14:59.538097 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:14:59.538054    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:15:00.121161 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:00.121121    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 08 07:15:00.126583 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:00.126560    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 07:15:00.128155 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:00.128131    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:15:04.542881 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:04.542855    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:15:04.543354 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:04.543325    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:15:05.121196 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:05.121152    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 08 07:15:05.121371 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:05.121280    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:15:10.121805 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:10.121735    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 08 07:15:10.127077 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:10.127052    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 07:15:10.128579 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:10.128562    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:15:14.543972 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:14.543931    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:15:15.121982 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:15.121942    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 08 07:15:20.121556 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:20.121473    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 08 07:15:20.126837 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:20.126813    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 08 07:15:20.126967 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:20.126956    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:15:20.128375 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:20.128351    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 07:15:20.128454 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:20.128444    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:15:22.537567 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.537545    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:15:22.626866 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.626835    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kube-rbac-proxy-sar-config\") pod \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") "
Jul 08 07:15:22.627063 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.626887    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-proxy-tls\") pod \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") "
Jul 08 07:15:22.627063 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.626906    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-cabundle-cert\") pod \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") "
Jul 08 07:15:22.627063 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.626922    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vfzd\" (UniqueName: \"kubernetes.io/projected/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kube-api-access-8vfzd\") pod \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") "
Jul 08 07:15:22.627063 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.626950    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kserve-provision-location\") pod \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\" (UID: \"3dce2faa-25c2-4a73-9fc2-75885d71fd8e\") "
Jul 08 07:15:22.627292 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.627265    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3dce2faa-25c2-4a73-9fc2-75885d71fd8e" (UID: "3dce2faa-25c2-4a73-9fc2-75885d71fd8e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:15:22.627344 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.627332    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3dce2faa-25c2-4a73-9fc2-75885d71fd8e" (UID: "3dce2faa-25c2-4a73-9fc2-75885d71fd8e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:15:22.627447 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.627419    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3dce2faa-25c2-4a73-9fc2-75885d71fd8e" (UID: "3dce2faa-25c2-4a73-9fc2-75885d71fd8e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:15:22.629192 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.629160    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kube-api-access-8vfzd" (OuterVolumeSpecName: "kube-api-access-8vfzd") pod "3dce2faa-25c2-4a73-9fc2-75885d71fd8e" (UID: "3dce2faa-25c2-4a73-9fc2-75885d71fd8e"). InnerVolumeSpecName "kube-api-access-8vfzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:15:22.629307 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.629208    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3dce2faa-25c2-4a73-9fc2-75885d71fd8e" (UID: "3dce2faa-25c2-4a73-9fc2-75885d71fd8e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:15:22.642713 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.642686    2557 generic.go:356] "Generic (PLEG): container finished" podID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerID="1dd38d765c3a0dda6d6eec460cbdedb4ccb6699163d7f59e51bc9bce05ca3aa0" exitCode=0
Jul 08 07:15:22.642852 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.642742    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"
Jul 08 07:15:22.642852 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.642750    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" event={"ID":"3dce2faa-25c2-4a73-9fc2-75885d71fd8e","Type":"ContainerDied","Data":"1dd38d765c3a0dda6d6eec460cbdedb4ccb6699163d7f59e51bc9bce05ca3aa0"}
Jul 08 07:15:22.642852 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.642838    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp" event={"ID":"3dce2faa-25c2-4a73-9fc2-75885d71fd8e","Type":"ContainerDied","Data":"4a81d1fbbf611d336c81dfac07fa9e5a8d8863043b0502977a80f53de44966a1"}
Jul 08 07:15:22.643023 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.642884    2557 scope.go:117] "RemoveContainer" containerID="1dd38d765c3a0dda6d6eec460cbdedb4ccb6699163d7f59e51bc9bce05ca3aa0"
Jul 08 07:15:22.650936 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.650917    2557 scope.go:117] "RemoveContainer" containerID="c71089ec4b0cbc3d67a2e7f646dc65c26402dfa2cd611630cee43a257a6f55ca"
Jul 08 07:15:22.658072 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.658057    2557 scope.go:117] "RemoveContainer" containerID="83793913ee055a274ed64ea9b0a717ab53dd8e0d75f85bb929ef6e28c7892eb6"
Jul 08 07:15:22.664401 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.664381    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"]
Jul 08 07:15:22.665252 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.665236    2557 scope.go:117] "RemoveContainer" containerID="7dc159f698b4d083f315679ca7612d750b669de26d7029d5c813a020940ac12d"
Jul 08 07:15:22.668247 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.668227    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-53e83-predictor-6f4c9d7d48-xt2lp"]
Jul 08 07:15:22.672807 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.672791    2557 scope.go:117] "RemoveContainer" containerID="1dd38d765c3a0dda6d6eec460cbdedb4ccb6699163d7f59e51bc9bce05ca3aa0"
Jul 08 07:15:22.673043 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:15:22.673026    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd38d765c3a0dda6d6eec460cbdedb4ccb6699163d7f59e51bc9bce05ca3aa0\": container with ID starting with 1dd38d765c3a0dda6d6eec460cbdedb4ccb6699163d7f59e51bc9bce05ca3aa0 not found: ID does not exist" containerID="1dd38d765c3a0dda6d6eec460cbdedb4ccb6699163d7f59e51bc9bce05ca3aa0"
Jul 08 07:15:22.673092 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.673050    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd38d765c3a0dda6d6eec460cbdedb4ccb6699163d7f59e51bc9bce05ca3aa0"} err="failed to get container status \"1dd38d765c3a0dda6d6eec460cbdedb4ccb6699163d7f59e51bc9bce05ca3aa0\": rpc error: code = NotFound desc = could not find container \"1dd38d765c3a0dda6d6eec460cbdedb4ccb6699163d7f59e51bc9bce05ca3aa0\": container with ID starting with 1dd38d765c3a0dda6d6eec460cbdedb4ccb6699163d7f59e51bc9bce05ca3aa0 not found: ID does not exist"
Jul 08 07:15:22.673092 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.673065    2557 scope.go:117] "RemoveContainer" containerID="c71089ec4b0cbc3d67a2e7f646dc65c26402dfa2cd611630cee43a257a6f55ca"
Jul 08 07:15:22.673302 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:15:22.673285    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71089ec4b0cbc3d67a2e7f646dc65c26402dfa2cd611630cee43a257a6f55ca\": container with ID starting with c71089ec4b0cbc3d67a2e7f646dc65c26402dfa2cd611630cee43a257a6f55ca not found: ID does not exist" containerID="c71089ec4b0cbc3d67a2e7f646dc65c26402dfa2cd611630cee43a257a6f55ca"
Jul 08 07:15:22.673339 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.673308    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71089ec4b0cbc3d67a2e7f646dc65c26402dfa2cd611630cee43a257a6f55ca"} err="failed to get container status \"c71089ec4b0cbc3d67a2e7f646dc65c26402dfa2cd611630cee43a257a6f55ca\": rpc error: code = NotFound desc = could not find container \"c71089ec4b0cbc3d67a2e7f646dc65c26402dfa2cd611630cee43a257a6f55ca\": container with ID starting with c71089ec4b0cbc3d67a2e7f646dc65c26402dfa2cd611630cee43a257a6f55ca not found: ID does not exist"
Jul 08 07:15:22.673339 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.673325    2557 scope.go:117] "RemoveContainer" containerID="83793913ee055a274ed64ea9b0a717ab53dd8e0d75f85bb929ef6e28c7892eb6"
Jul 08 07:15:22.673603 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:15:22.673587    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83793913ee055a274ed64ea9b0a717ab53dd8e0d75f85bb929ef6e28c7892eb6\": container with ID starting with 83793913ee055a274ed64ea9b0a717ab53dd8e0d75f85bb929ef6e28c7892eb6 not found: ID does not exist" containerID="83793913ee055a274ed64ea9b0a717ab53dd8e0d75f85bb929ef6e28c7892eb6"
Jul 08 07:15:22.673640 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.673617    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83793913ee055a274ed64ea9b0a717ab53dd8e0d75f85bb929ef6e28c7892eb6"} err="failed to get container status \"83793913ee055a274ed64ea9b0a717ab53dd8e0d75f85bb929ef6e28c7892eb6\": rpc error: code = NotFound desc = could not find container \"83793913ee055a274ed64ea9b0a717ab53dd8e0d75f85bb929ef6e28c7892eb6\": container with ID starting with 83793913ee055a274ed64ea9b0a717ab53dd8e0d75f85bb929ef6e28c7892eb6 not found: ID does not exist"
Jul 08 07:15:22.673640 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.673632    2557 scope.go:117] "RemoveContainer" containerID="7dc159f698b4d083f315679ca7612d750b669de26d7029d5c813a020940ac12d"
Jul 08 07:15:22.673875 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:15:22.673859    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc159f698b4d083f315679ca7612d750b669de26d7029d5c813a020940ac12d\": container with ID starting with 7dc159f698b4d083f315679ca7612d750b669de26d7029d5c813a020940ac12d not found: ID does not exist" containerID="7dc159f698b4d083f315679ca7612d750b669de26d7029d5c813a020940ac12d"
Jul 08 07:15:22.673924 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.673881    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc159f698b4d083f315679ca7612d750b669de26d7029d5c813a020940ac12d"} err="failed to get container status \"7dc159f698b4d083f315679ca7612d750b669de26d7029d5c813a020940ac12d\": rpc error: code = NotFound desc = could not find container \"7dc159f698b4d083f315679ca7612d750b669de26d7029d5c813a020940ac12d\": container with ID starting with 7dc159f698b4d083f315679ca7612d750b669de26d7029d5c813a020940ac12d not found: ID does not exist"
Jul 08 07:15:22.727673 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.727655    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kserve-provision-location\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:15:22.727673 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.727675    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:15:22.727825 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.727684    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-proxy-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:15:22.727825 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.727693    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-cabundle-cert\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:15:22.727825 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:22.727701    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vfzd\" (UniqueName: \"kubernetes.io/projected/3dce2faa-25c2-4a73-9fc2-75885d71fd8e-kube-api-access-8vfzd\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:15:23.787648 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:23.787613    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" path="/var/lib/kubelet/pods/3dce2faa-25c2-4a73-9fc2-75885d71fd8e/volumes"
Jul 08 07:15:24.543594 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:24.543551    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:15:34.543278 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:34.543241    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:15:44.543797 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:44.543739    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:15:54.544178 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:15:54.544133    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:16:04.543304 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:16:04.543264    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:16:14.543723 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:16:14.543684    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:16:24.544093 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:16:24.544050    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:16:28.781170 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:16:28.781128    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:16:38.781524 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:16:38.781479    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:16:48.781195 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:16:48.781159    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:16:58.781836 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:16:58.781794    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:17:08.781486 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:08.781440    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:17:18.781720 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:18.781683    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:17:28.781798 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:28.781743    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:17:32.570378 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:32.570344    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"]
Jul 08 07:17:32.570846 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:32.570608    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" containerID="cri-o://95b51fa31ca8fd84d7db4e059d65146b96bf415622bdc5ab87942b2f2265aa81" gracePeriod=30
Jul 08 07:17:32.570846 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:32.570693    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kube-rbac-proxy" containerID="cri-o://f5690e1a0841de90a2b75151e93d7982586d263359395530b6e1062878e31157" gracePeriod=30
Jul 08 07:17:33.161218 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:33.161185    2557 generic.go:356] "Generic (PLEG): container finished" podID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerID="f5690e1a0841de90a2b75151e93d7982586d263359395530b6e1062878e31157" exitCode=2
Jul 08 07:17:33.161372 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:33.161260    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" event={"ID":"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25","Type":"ContainerDied","Data":"f5690e1a0841de90a2b75151e93d7982586d263359395530b6e1062878e31157"}
Jul 08 07:17:34.538955 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:34.538906    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 08 07:17:37.730876 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.730845    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"]
Jul 08 07:17:37.731236 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731182    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kube-rbac-proxy"
Jul 08 07:17:37.731236 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731195    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kube-rbac-proxy"
Jul 08 07:17:37.731236 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731210    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent"
Jul 08 07:17:37.731236 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731216    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent"
Jul 08 07:17:37.731236 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731221    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e94490-b45d-4292-baf1-0aa81f223a21" containerName="kserve-container"
Jul 08 07:17:37.731236 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731228    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e94490-b45d-4292-baf1-0aa81f223a21" containerName="kserve-container"
Jul 08 07:17:37.731448 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731243    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e94490-b45d-4292-baf1-0aa81f223a21" containerName="kube-rbac-proxy"
Jul 08 07:17:37.731448 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731249    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e94490-b45d-4292-baf1-0aa81f223a21" containerName="kube-rbac-proxy"
Jul 08 07:17:37.731448 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731257    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="storage-initializer"
Jul 08 07:17:37.731448 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731262    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="storage-initializer"
Jul 08 07:17:37.731448 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731272    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container"
Jul 08 07:17:37.731448 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731277    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container"
Jul 08 07:17:37.731448 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731332    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kube-rbac-proxy"
Jul 08 07:17:37.731448 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731343    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4e94490-b45d-4292-baf1-0aa81f223a21" containerName="kserve-container"
Jul 08 07:17:37.731448 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731349    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="kserve-container"
Jul 08 07:17:37.731448 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731355    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4e94490-b45d-4292-baf1-0aa81f223a21" containerName="kube-rbac-proxy"
Jul 08 07:17:37.731448 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.731360    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dce2faa-25c2-4a73-9fc2-75885d71fd8e" containerName="agent"
Jul 08 07:17:37.734718 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.734699    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.737713 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.737692    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-f84f3e-predictor-serving-cert\""
Jul 08 07:17:37.737839 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.737820    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-f84f3e-kube-rbac-proxy-sar-config\""
Jul 08 07:17:37.756379 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.756356    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"]
Jul 08 07:17:37.803112 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.803087    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb4lq\" (UniqueName: \"kubernetes.io/projected/d1397740-8702-41ff-a059-a667771fde92-kube-api-access-nb4lq\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.803222 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.803117    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1397740-8702-41ff-a059-a667771fde92-proxy-tls\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.803222 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.803147    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1397740-8702-41ff-a059-a667771fde92-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.803222 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.803199    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1397740-8702-41ff-a059-a667771fde92-kserve-provision-location\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.803222 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.803216    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1397740-8702-41ff-a059-a667771fde92-cabundle-cert\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.903631 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.903594    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1397740-8702-41ff-a059-a667771fde92-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.903782 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.903656    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1397740-8702-41ff-a059-a667771fde92-kserve-provision-location\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.903782 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.903676    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1397740-8702-41ff-a059-a667771fde92-cabundle-cert\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.903782 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.903703    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb4lq\" (UniqueName: \"kubernetes.io/projected/d1397740-8702-41ff-a059-a667771fde92-kube-api-access-nb4lq\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.903782 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.903723    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1397740-8702-41ff-a059-a667771fde92-proxy-tls\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.904185 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.904163    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1397740-8702-41ff-a059-a667771fde92-kserve-provision-location\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.904349 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.904328    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1397740-8702-41ff-a059-a667771fde92-kube-rbac-proxy-sar-config\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.904397 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.904382    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1397740-8702-41ff-a059-a667771fde92-cabundle-cert\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.906142 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.906123    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1397740-8702-41ff-a059-a667771fde92-proxy-tls\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:37.912939 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:37.912919    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb4lq\" (UniqueName: \"kubernetes.io/projected/d1397740-8702-41ff-a059-a667771fde92-kube-api-access-nb4lq\") pod \"isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") " pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:38.044501 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:38.044449    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:38.174075 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:38.174052    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"]
Jul 08 07:17:38.176455 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:17:38.176416    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1397740_8702_41ff_a059_a667771fde92.slice/crio-ec8de1863fa451cc8ff54d28e13b14b57f3e797d0e9b055a2c3a206d1f690f26 WatchSource:0}: Error finding container ec8de1863fa451cc8ff54d28e13b14b57f3e797d0e9b055a2c3a206d1f690f26: Status 404 returned error can't find the container with id ec8de1863fa451cc8ff54d28e13b14b57f3e797d0e9b055a2c3a206d1f690f26
Jul 08 07:17:38.182096 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:38.182066    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" event={"ID":"d1397740-8702-41ff-a059-a667771fde92","Type":"ContainerStarted","Data":"ec8de1863fa451cc8ff54d28e13b14b57f3e797d0e9b055a2c3a206d1f690f26"}
Jul 08 07:17:38.781243 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:38.781203    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 08 07:17:39.187900 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:39.187819    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" event={"ID":"d1397740-8702-41ff-a059-a667771fde92","Type":"ContainerStarted","Data":"623eae6f246e077e7046f3dfea75868df00ea060dab42e6b8bef108fbf99930c"}
Jul 08 07:17:39.539220 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:39.539183    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 08 07:17:41.512142 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.512117    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:17:41.529654 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.529632    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-proxy-tls\") pod \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") "
Jul 08 07:17:41.529798 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.529675    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6pd7\" (UniqueName: \"kubernetes.io/projected/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kube-api-access-m6pd7\") pod \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") "
Jul 08 07:17:41.529798 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.529701    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-cabundle-cert\") pod \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") "
Jul 08 07:17:41.529882 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.529809    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kube-rbac-proxy-sar-config\") pod \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") "
Jul 08 07:17:41.529929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.529899    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kserve-provision-location\") pod \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\" (UID: \"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25\") "
Jul 08 07:17:41.530008 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.529981    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" (UID: "395bc3c1-2bdc-4dbd-aa3f-82ef64696d25"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:17:41.530141 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.530125    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-cabundle-cert\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:17:41.530215 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.530155    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" (UID: "395bc3c1-2bdc-4dbd-aa3f-82ef64696d25"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:17:41.530277 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.530214    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" (UID: "395bc3c1-2bdc-4dbd-aa3f-82ef64696d25"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:17:41.532145 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.532112    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kube-api-access-m6pd7" (OuterVolumeSpecName: "kube-api-access-m6pd7") pod "395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" (UID: "395bc3c1-2bdc-4dbd-aa3f-82ef64696d25"). InnerVolumeSpecName "kube-api-access-m6pd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:17:41.532253 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.532141    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" (UID: "395bc3c1-2bdc-4dbd-aa3f-82ef64696d25"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:17:41.631231 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.631175    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6pd7\" (UniqueName: \"kubernetes.io/projected/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kube-api-access-m6pd7\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:17:41.631231 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.631198    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:17:41.631231 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.631207    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-kserve-provision-location\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:17:41.631231 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:41.631216    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25-proxy-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:17:42.200003 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.199972    2557 generic.go:356] "Generic (PLEG): container finished" podID="d1397740-8702-41ff-a059-a667771fde92" containerID="623eae6f246e077e7046f3dfea75868df00ea060dab42e6b8bef108fbf99930c" exitCode=0
Jul 08 07:17:42.200140 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.200049    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" event={"ID":"d1397740-8702-41ff-a059-a667771fde92","Type":"ContainerDied","Data":"623eae6f246e077e7046f3dfea75868df00ea060dab42e6b8bef108fbf99930c"}
Jul 08 07:17:42.201856 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.201832    2557 generic.go:356] "Generic (PLEG): container finished" podID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerID="95b51fa31ca8fd84d7db4e059d65146b96bf415622bdc5ab87942b2f2265aa81" exitCode=0
Jul 08 07:17:42.201950 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.201872    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" event={"ID":"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25","Type":"ContainerDied","Data":"95b51fa31ca8fd84d7db4e059d65146b96bf415622bdc5ab87942b2f2265aa81"}
Jul 08 07:17:42.201950 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.201885    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"
Jul 08 07:17:42.201950 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.201913    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp" event={"ID":"395bc3c1-2bdc-4dbd-aa3f-82ef64696d25","Type":"ContainerDied","Data":"247aace483389ee3cc408f44970b1146e5d2cbfc660f897bdbc6e57df3bcf449"}
Jul 08 07:17:42.201950 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.201941    2557 scope.go:117] "RemoveContainer" containerID="f5690e1a0841de90a2b75151e93d7982586d263359395530b6e1062878e31157"
Jul 08 07:17:42.211641 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.211620    2557 scope.go:117] "RemoveContainer" containerID="95b51fa31ca8fd84d7db4e059d65146b96bf415622bdc5ab87942b2f2265aa81"
Jul 08 07:17:42.220898 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.220880    2557 scope.go:117] "RemoveContainer" containerID="5c18ac7166e1b3ae03a599276d8cd187e33801d4d23632643855ff51ade264d5"
Jul 08 07:17:42.232769 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.232740    2557 scope.go:117] "RemoveContainer" containerID="f5690e1a0841de90a2b75151e93d7982586d263359395530b6e1062878e31157"
Jul 08 07:17:42.233148 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:17:42.233125    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5690e1a0841de90a2b75151e93d7982586d263359395530b6e1062878e31157\": container with ID starting with f5690e1a0841de90a2b75151e93d7982586d263359395530b6e1062878e31157 not found: ID does not exist" containerID="f5690e1a0841de90a2b75151e93d7982586d263359395530b6e1062878e31157"
Jul 08 07:17:42.233252 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.233156    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5690e1a0841de90a2b75151e93d7982586d263359395530b6e1062878e31157"} err="failed to get container status \"f5690e1a0841de90a2b75151e93d7982586d263359395530b6e1062878e31157\": rpc error: code = NotFound desc = could not find container \"f5690e1a0841de90a2b75151e93d7982586d263359395530b6e1062878e31157\": container with ID starting with f5690e1a0841de90a2b75151e93d7982586d263359395530b6e1062878e31157 not found: ID does not exist"
Jul 08 07:17:42.233252 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.233178    2557 scope.go:117] "RemoveContainer" containerID="95b51fa31ca8fd84d7db4e059d65146b96bf415622bdc5ab87942b2f2265aa81"
Jul 08 07:17:42.233488 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:17:42.233452    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b51fa31ca8fd84d7db4e059d65146b96bf415622bdc5ab87942b2f2265aa81\": container with ID starting with 95b51fa31ca8fd84d7db4e059d65146b96bf415622bdc5ab87942b2f2265aa81 not found: ID does not exist" containerID="95b51fa31ca8fd84d7db4e059d65146b96bf415622bdc5ab87942b2f2265aa81"
Jul 08 07:17:42.233568 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.233482    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b51fa31ca8fd84d7db4e059d65146b96bf415622bdc5ab87942b2f2265aa81"} err="failed to get container status \"95b51fa31ca8fd84d7db4e059d65146b96bf415622bdc5ab87942b2f2265aa81\": rpc error: code = NotFound desc = could not find container \"95b51fa31ca8fd84d7db4e059d65146b96bf415622bdc5ab87942b2f2265aa81\": container with ID starting with 95b51fa31ca8fd84d7db4e059d65146b96bf415622bdc5ab87942b2f2265aa81 not found: ID does not exist"
Jul 08 07:17:42.233568 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.233505    2557 scope.go:117] "RemoveContainer" containerID="5c18ac7166e1b3ae03a599276d8cd187e33801d4d23632643855ff51ade264d5"
Jul 08 07:17:42.234020 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:17:42.233996    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c18ac7166e1b3ae03a599276d8cd187e33801d4d23632643855ff51ade264d5\": container with ID starting with 5c18ac7166e1b3ae03a599276d8cd187e33801d4d23632643855ff51ade264d5 not found: ID does not exist" containerID="5c18ac7166e1b3ae03a599276d8cd187e33801d4d23632643855ff51ade264d5"
Jul 08 07:17:42.234084 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.234031    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c18ac7166e1b3ae03a599276d8cd187e33801d4d23632643855ff51ade264d5"} err="failed to get container status \"5c18ac7166e1b3ae03a599276d8cd187e33801d4d23632643855ff51ade264d5\": rpc error: code = NotFound desc = could not find container \"5c18ac7166e1b3ae03a599276d8cd187e33801d4d23632643855ff51ade264d5\": container with ID starting with 5c18ac7166e1b3ae03a599276d8cd187e33801d4d23632643855ff51ade264d5 not found: ID does not exist"
Jul 08 07:17:42.235306 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.235283    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"]
Jul 08 07:17:42.240464 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:42.240441    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-bba5b-predictor-66899b6f89-8jqwp"]
Jul 08 07:17:43.209003 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:43.208971    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" event={"ID":"d1397740-8702-41ff-a059-a667771fde92","Type":"ContainerStarted","Data":"20e89a5997d668c8f5e7a63bfdd15ad75eb9ae48e24fa6f581b7dbc3fc549c42"}
Jul 08 07:17:43.209411 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:43.209038    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" event={"ID":"d1397740-8702-41ff-a059-a667771fde92","Type":"ContainerStarted","Data":"f634c866bfab173aa0770ef81a1e23f14cc9949640dd4e8c7b5618a2fad7fefc"}
Jul 08 07:17:43.209411 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:43.209385    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:43.209519 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:43.209434    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:43.210556 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:43.210532    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 07:17:43.228908 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:43.228869    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" podStartSLOduration=2.4491720040000002 podStartE2EDuration="6.22885447s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.779682466s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:17:43.227356767 +0000 UTC m=+1395.963701482" watchObservedRunningTime="2026-07-08 07:17:43.22885447 +0000 UTC m=+1395.965199190"
Jul 08 07:17:43.784818 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:43.784784    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" path="/var/lib/kubelet/pods/395bc3c1-2bdc-4dbd-aa3f-82ef64696d25/volumes"
Jul 08 07:17:44.214783 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:44.214673    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 07:17:49.219793 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:49.219744    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:17:49.220349 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:49.220317    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 07:17:59.220703 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:17:59.220663    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 07:18:09.220282 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:09.220241    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 07:18:19.220274 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:19.220190    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 07:18:29.220714 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:29.220673    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 07:18:39.220661 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:39.220568    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 07:18:49.220972 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:49.220929    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 08 07:18:52.781893 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:52.781857    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:18:57.855748 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.855716    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"]
Jul 08 07:18:57.856114 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.856095    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="storage-initializer"
Jul 08 07:18:57.856160 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.856114    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="storage-initializer"
Jul 08 07:18:57.856160 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.856126    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kube-rbac-proxy"
Jul 08 07:18:57.856160 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.856135    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kube-rbac-proxy"
Jul 08 07:18:57.856160 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.856151    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container"
Jul 08 07:18:57.856283 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.856160    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container"
Jul 08 07:18:57.856283 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.856244    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kserve-container"
Jul 08 07:18:57.856283 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.856254    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="395bc3c1-2bdc-4dbd-aa3f-82ef64696d25" containerName="kube-rbac-proxy"
Jul 08 07:18:57.859283 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.859268    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:57.861629 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.861610    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-f84f3e\""
Jul 08 07:18:57.861890 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.861863    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-f84f3e-predictor-serving-cert\""
Jul 08 07:18:57.861890 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.861863    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-f84f3e-kube-rbac-proxy-sar-config\""
Jul 08 07:18:57.862065 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.861933    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-f84f3e-dockercfg-2vb97\""
Jul 08 07:18:57.869532 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.869511    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"]
Jul 08 07:18:57.923558 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.923529    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw89w\" (UniqueName: \"kubernetes.io/projected/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kube-api-access-pw89w\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:57.923689 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.923577    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-proxy-tls\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:57.923689 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.923592    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-cabundle-cert\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:57.923689 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.923612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kserve-provision-location\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:57.923845 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:57.923712    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:58.025131 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:58.025097    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-proxy-tls\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:58.025131 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:58.025131    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-cabundle-cert\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:58.025358 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:58.025155    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kserve-provision-location\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:58.025358 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:58.025208    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:58.025358 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:58.025250    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw89w\" (UniqueName: \"kubernetes.io/projected/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kube-api-access-pw89w\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:58.025718 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:58.025695    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kserve-provision-location\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:58.025918 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:58.025899    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-cabundle-cert\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:58.025985 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:58.025908    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:58.027737 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:58.027716    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-proxy-tls\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:58.033202 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:58.033180    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw89w\" (UniqueName: \"kubernetes.io/projected/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kube-api-access-pw89w\") pod \"isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") " pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:58.169631 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:58.169566    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:18:58.296911 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:58.296846    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"]
Jul 08 07:18:58.299068 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:18:58.299035    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2769bb4f_9abd_465e_a9d3_c68dbf3feabf.slice/crio-85f6558c0a75b31cd0934821b65990e8e1cdadf911539ffe2820d83c1b50ce2b WatchSource:0}: Error finding container 85f6558c0a75b31cd0934821b65990e8e1cdadf911539ffe2820d83c1b50ce2b: Status 404 returned error can't find the container with id 85f6558c0a75b31cd0934821b65990e8e1cdadf911539ffe2820d83c1b50ce2b
Jul 08 07:18:58.519209 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:58.519171    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr" event={"ID":"2769bb4f-9abd-465e-a9d3-c68dbf3feabf","Type":"ContainerStarted","Data":"6a4d606443d46c9df80a060a2ea3a0e90edb44232c6417a7bf69c6f7e113e990"}
Jul 08 07:18:58.519401 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:18:58.519231    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr" event={"ID":"2769bb4f-9abd-465e-a9d3-c68dbf3feabf","Type":"ContainerStarted","Data":"85f6558c0a75b31cd0934821b65990e8e1cdadf911539ffe2820d83c1b50ce2b"}
Jul 08 07:19:03.541626 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:03.541595    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr_2769bb4f-9abd-465e-a9d3-c68dbf3feabf/storage-initializer/0.log"
Jul 08 07:19:03.542073 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:03.541638    2557 generic.go:356] "Generic (PLEG): container finished" podID="2769bb4f-9abd-465e-a9d3-c68dbf3feabf" containerID="6a4d606443d46c9df80a060a2ea3a0e90edb44232c6417a7bf69c6f7e113e990" exitCode=1
Jul 08 07:19:03.542073 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:03.541664    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr" event={"ID":"2769bb4f-9abd-465e-a9d3-c68dbf3feabf","Type":"ContainerDied","Data":"6a4d606443d46c9df80a060a2ea3a0e90edb44232c6417a7bf69c6f7e113e990"}
Jul 08 07:19:04.548022 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:04.547994    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr_2769bb4f-9abd-465e-a9d3-c68dbf3feabf/storage-initializer/0.log"
Jul 08 07:19:04.548392 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:04.548082    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr" event={"ID":"2769bb4f-9abd-465e-a9d3-c68dbf3feabf","Type":"ContainerStarted","Data":"644032cd05213c668303764539a0a11e6c91850849b9ec55954c6d0bc608ff29"}
Jul 08 07:19:06.557306 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:06.557278    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr_2769bb4f-9abd-465e-a9d3-c68dbf3feabf/storage-initializer/1.log"
Jul 08 07:19:06.557726 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:06.557645    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr_2769bb4f-9abd-465e-a9d3-c68dbf3feabf/storage-initializer/0.log"
Jul 08 07:19:06.557726 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:06.557677    2557 generic.go:356] "Generic (PLEG): container finished" podID="2769bb4f-9abd-465e-a9d3-c68dbf3feabf" containerID="644032cd05213c668303764539a0a11e6c91850849b9ec55954c6d0bc608ff29" exitCode=1
Jul 08 07:19:06.557726 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:06.557702    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr" event={"ID":"2769bb4f-9abd-465e-a9d3-c68dbf3feabf","Type":"ContainerDied","Data":"644032cd05213c668303764539a0a11e6c91850849b9ec55954c6d0bc608ff29"}
Jul 08 07:19:06.557872 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:06.557773    2557 scope.go:117] "RemoveContainer" containerID="6a4d606443d46c9df80a060a2ea3a0e90edb44232c6417a7bf69c6f7e113e990"
Jul 08 07:19:06.558168 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:06.558134    2557 scope.go:117] "RemoveContainer" containerID="6a4d606443d46c9df80a060a2ea3a0e90edb44232c6417a7bf69c6f7e113e990"
Jul 08 07:19:06.568856 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:19:06.568831    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr_kserve-ci-e2e-test(2769bb4f-9abd-465e-a9d3-c68dbf3feabf)\"" pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr" podUID="2769bb4f-9abd-465e-a9d3-c68dbf3feabf"
Jul 08 07:19:06.568983 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:19:06.568956    2557 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr_kserve-ci-e2e-test_2769bb4f-9abd-465e-a9d3-c68dbf3feabf_0 in pod sandbox 85f6558c0a75b31cd0934821b65990e8e1cdadf911539ffe2820d83c1b50ce2b from index: no such id: '6a4d606443d46c9df80a060a2ea3a0e90edb44232c6417a7bf69c6f7e113e990'" containerID="6a4d606443d46c9df80a060a2ea3a0e90edb44232c6417a7bf69c6f7e113e990"
Jul 08 07:19:06.569038 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:06.568993    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4d606443d46c9df80a060a2ea3a0e90edb44232c6417a7bf69c6f7e113e990"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr_kserve-ci-e2e-test_2769bb4f-9abd-465e-a9d3-c68dbf3feabf_0 in pod sandbox 85f6558c0a75b31cd0934821b65990e8e1cdadf911539ffe2820d83c1b50ce2b from index: no such id: '6a4d606443d46c9df80a060a2ea3a0e90edb44232c6417a7bf69c6f7e113e990'"
Jul 08 07:19:07.562791 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:07.562745    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr_2769bb4f-9abd-465e-a9d3-c68dbf3feabf/storage-initializer/1.log"
Jul 08 07:19:13.904886 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:13.904854    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"]
Jul 08 07:19:13.968090 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:13.968059    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"]
Jul 08 07:19:13.968425 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:13.968359    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kserve-container" containerID="cri-o://f634c866bfab173aa0770ef81a1e23f14cc9949640dd4e8c7b5618a2fad7fefc" gracePeriod=30
Jul 08 07:19:13.968621 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:13.968561    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kube-rbac-proxy" containerID="cri-o://20e89a5997d668c8f5e7a63bfdd15ad75eb9ae48e24fa6f581b7dbc3fc549c42" gracePeriod=30
Jul 08 07:19:14.088099 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.088080    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr_2769bb4f-9abd-465e-a9d3-c68dbf3feabf/storage-initializer/1.log"
Jul 08 07:19:14.090183 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.090169    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:19:14.153378 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.153347    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw89w\" (UniqueName: \"kubernetes.io/projected/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kube-api-access-pw89w\") pod \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") "
Jul 08 07:19:14.153531 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.153407    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kserve-provision-location\") pod \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") "
Jul 08 07:19:14.153531 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.153475    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-cabundle-cert\") pod \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") "
Jul 08 07:19:14.153531 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.153510    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kube-rbac-proxy-sar-config\") pod \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") "
Jul 08 07:19:14.153685 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.153535    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-proxy-tls\") pod \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\" (UID: \"2769bb4f-9abd-465e-a9d3-c68dbf3feabf\") "
Jul 08 07:19:14.153736 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.153671    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2769bb4f-9abd-465e-a9d3-c68dbf3feabf" (UID: "2769bb4f-9abd-465e-a9d3-c68dbf3feabf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:19:14.153843 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.153828    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kserve-provision-location\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:14.153897 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.153847    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2769bb4f-9abd-465e-a9d3-c68dbf3feabf" (UID: "2769bb4f-9abd-465e-a9d3-c68dbf3feabf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:19:14.153933 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.153907    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2769bb4f-9abd-465e-a9d3-c68dbf3feabf" (UID: "2769bb4f-9abd-465e-a9d3-c68dbf3feabf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:19:14.155640 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.155579    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kube-api-access-pw89w" (OuterVolumeSpecName: "kube-api-access-pw89w") pod "2769bb4f-9abd-465e-a9d3-c68dbf3feabf" (UID: "2769bb4f-9abd-465e-a9d3-c68dbf3feabf"). InnerVolumeSpecName "kube-api-access-pw89w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:19:14.155640 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.155613    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2769bb4f-9abd-465e-a9d3-c68dbf3feabf" (UID: "2769bb4f-9abd-465e-a9d3-c68dbf3feabf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:19:14.214992 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.214959    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 08 07:19:14.254693 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.254670    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:14.254693 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.254690    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-proxy-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:14.254851 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.254699    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw89w\" (UniqueName: \"kubernetes.io/projected/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-kube-api-access-pw89w\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:14.254851 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.254707    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2769bb4f-9abd-465e-a9d3-c68dbf3feabf-cabundle-cert\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:14.591638 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.591612    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr_2769bb4f-9abd-465e-a9d3-c68dbf3feabf/storage-initializer/1.log"
Jul 08 07:19:14.591835 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.591710    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr" event={"ID":"2769bb4f-9abd-465e-a9d3-c68dbf3feabf","Type":"ContainerDied","Data":"85f6558c0a75b31cd0934821b65990e8e1cdadf911539ffe2820d83c1b50ce2b"}
Jul 08 07:19:14.591835 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.591717    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"
Jul 08 07:19:14.591835 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.591791    2557 scope.go:117] "RemoveContainer" containerID="644032cd05213c668303764539a0a11e6c91850849b9ec55954c6d0bc608ff29"
Jul 08 07:19:14.600671 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.600637    2557 generic.go:356] "Generic (PLEG): container finished" podID="d1397740-8702-41ff-a059-a667771fde92" containerID="20e89a5997d668c8f5e7a63bfdd15ad75eb9ae48e24fa6f581b7dbc3fc549c42" exitCode=2
Jul 08 07:19:14.600803 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.600709    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" event={"ID":"d1397740-8702-41ff-a059-a667771fde92","Type":"ContainerDied","Data":"20e89a5997d668c8f5e7a63bfdd15ad75eb9ae48e24fa6f581b7dbc3fc549c42"}
Jul 08 07:19:14.630337 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.630306    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"]
Jul 08 07:19:14.635850 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:14.635830    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-f84f3e-predictor-64f457bbbf-8f9zr"]
Jul 08 07:19:15.785438 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:15.785405    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2769bb4f-9abd-465e-a9d3-c68dbf3feabf" path="/var/lib/kubelet/pods/2769bb4f-9abd-465e-a9d3-c68dbf3feabf/volumes"
Jul 08 07:19:18.209337 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.209315    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:19:18.284783 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.284740    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1397740-8702-41ff-a059-a667771fde92-kube-rbac-proxy-sar-config\") pod \"d1397740-8702-41ff-a059-a667771fde92\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") "
Jul 08 07:19:18.284936 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.284795    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1397740-8702-41ff-a059-a667771fde92-kserve-provision-location\") pod \"d1397740-8702-41ff-a059-a667771fde92\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") "
Jul 08 07:19:18.284936 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.284863    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb4lq\" (UniqueName: \"kubernetes.io/projected/d1397740-8702-41ff-a059-a667771fde92-kube-api-access-nb4lq\") pod \"d1397740-8702-41ff-a059-a667771fde92\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") "
Jul 08 07:19:18.284936 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.284909    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1397740-8702-41ff-a059-a667771fde92-proxy-tls\") pod \"d1397740-8702-41ff-a059-a667771fde92\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") "
Jul 08 07:19:18.285106 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.284941    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1397740-8702-41ff-a059-a667771fde92-cabundle-cert\") pod \"d1397740-8702-41ff-a059-a667771fde92\" (UID: \"d1397740-8702-41ff-a059-a667771fde92\") "
Jul 08 07:19:18.285167 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.285106    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1397740-8702-41ff-a059-a667771fde92-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1397740-8702-41ff-a059-a667771fde92" (UID: "d1397740-8702-41ff-a059-a667771fde92"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:19:18.285220 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.285168    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1397740-8702-41ff-a059-a667771fde92-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d1397740-8702-41ff-a059-a667771fde92" (UID: "d1397740-8702-41ff-a059-a667771fde92"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:19:18.285349 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.285325    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1397740-8702-41ff-a059-a667771fde92-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d1397740-8702-41ff-a059-a667771fde92" (UID: "d1397740-8702-41ff-a059-a667771fde92"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:19:18.287092 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.287073    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1397740-8702-41ff-a059-a667771fde92-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1397740-8702-41ff-a059-a667771fde92" (UID: "d1397740-8702-41ff-a059-a667771fde92"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:19:18.287234 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.287216    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1397740-8702-41ff-a059-a667771fde92-kube-api-access-nb4lq" (OuterVolumeSpecName: "kube-api-access-nb4lq") pod "d1397740-8702-41ff-a059-a667771fde92" (UID: "d1397740-8702-41ff-a059-a667771fde92"). InnerVolumeSpecName "kube-api-access-nb4lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:19:18.386411 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.386356    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1397740-8702-41ff-a059-a667771fde92-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:18.386411 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.386379    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1397740-8702-41ff-a059-a667771fde92-kserve-provision-location\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:18.386411 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.386390    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb4lq\" (UniqueName: \"kubernetes.io/projected/d1397740-8702-41ff-a059-a667771fde92-kube-api-access-nb4lq\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:18.386411 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.386399    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1397740-8702-41ff-a059-a667771fde92-proxy-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:18.386411 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.386408    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1397740-8702-41ff-a059-a667771fde92-cabundle-cert\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:18.618849 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.618813    2557 generic.go:356] "Generic (PLEG): container finished" podID="d1397740-8702-41ff-a059-a667771fde92" containerID="f634c866bfab173aa0770ef81a1e23f14cc9949640dd4e8c7b5618a2fad7fefc" exitCode=0
Jul 08 07:19:18.619005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.618866    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"
Jul 08 07:19:18.619005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.618901    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" event={"ID":"d1397740-8702-41ff-a059-a667771fde92","Type":"ContainerDied","Data":"f634c866bfab173aa0770ef81a1e23f14cc9949640dd4e8c7b5618a2fad7fefc"}
Jul 08 07:19:18.619005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.618968    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw" event={"ID":"d1397740-8702-41ff-a059-a667771fde92","Type":"ContainerDied","Data":"ec8de1863fa451cc8ff54d28e13b14b57f3e797d0e9b055a2c3a206d1f690f26"}
Jul 08 07:19:18.619005 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.618995    2557 scope.go:117] "RemoveContainer" containerID="20e89a5997d668c8f5e7a63bfdd15ad75eb9ae48e24fa6f581b7dbc3fc549c42"
Jul 08 07:19:18.627502 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.627479    2557 scope.go:117] "RemoveContainer" containerID="f634c866bfab173aa0770ef81a1e23f14cc9949640dd4e8c7b5618a2fad7fefc"
Jul 08 07:19:18.634708 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.634690    2557 scope.go:117] "RemoveContainer" containerID="623eae6f246e077e7046f3dfea75868df00ea060dab42e6b8bef108fbf99930c"
Jul 08 07:19:18.640888 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.640833    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"]
Jul 08 07:19:18.642236 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.642220    2557 scope.go:117] "RemoveContainer" containerID="20e89a5997d668c8f5e7a63bfdd15ad75eb9ae48e24fa6f581b7dbc3fc549c42"
Jul 08 07:19:18.642541 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:19:18.642517    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e89a5997d668c8f5e7a63bfdd15ad75eb9ae48e24fa6f581b7dbc3fc549c42\": container with ID starting with 20e89a5997d668c8f5e7a63bfdd15ad75eb9ae48e24fa6f581b7dbc3fc549c42 not found: ID does not exist" containerID="20e89a5997d668c8f5e7a63bfdd15ad75eb9ae48e24fa6f581b7dbc3fc549c42"
Jul 08 07:19:18.642681 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.642557    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e89a5997d668c8f5e7a63bfdd15ad75eb9ae48e24fa6f581b7dbc3fc549c42"} err="failed to get container status \"20e89a5997d668c8f5e7a63bfdd15ad75eb9ae48e24fa6f581b7dbc3fc549c42\": rpc error: code = NotFound desc = could not find container \"20e89a5997d668c8f5e7a63bfdd15ad75eb9ae48e24fa6f581b7dbc3fc549c42\": container with ID starting with 20e89a5997d668c8f5e7a63bfdd15ad75eb9ae48e24fa6f581b7dbc3fc549c42 not found: ID does not exist"
Jul 08 07:19:18.642681 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.642583    2557 scope.go:117] "RemoveContainer" containerID="f634c866bfab173aa0770ef81a1e23f14cc9949640dd4e8c7b5618a2fad7fefc"
Jul 08 07:19:18.643014 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:19:18.642979    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f634c866bfab173aa0770ef81a1e23f14cc9949640dd4e8c7b5618a2fad7fefc\": container with ID starting with f634c866bfab173aa0770ef81a1e23f14cc9949640dd4e8c7b5618a2fad7fefc not found: ID does not exist" containerID="f634c866bfab173aa0770ef81a1e23f14cc9949640dd4e8c7b5618a2fad7fefc"
Jul 08 07:19:18.643110 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.643022    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f634c866bfab173aa0770ef81a1e23f14cc9949640dd4e8c7b5618a2fad7fefc"} err="failed to get container status \"f634c866bfab173aa0770ef81a1e23f14cc9949640dd4e8c7b5618a2fad7fefc\": rpc error: code = NotFound desc = could not find container \"f634c866bfab173aa0770ef81a1e23f14cc9949640dd4e8c7b5618a2fad7fefc\": container with ID starting with f634c866bfab173aa0770ef81a1e23f14cc9949640dd4e8c7b5618a2fad7fefc not found: ID does not exist"
Jul 08 07:19:18.643110 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.643044    2557 scope.go:117] "RemoveContainer" containerID="623eae6f246e077e7046f3dfea75868df00ea060dab42e6b8bef108fbf99930c"
Jul 08 07:19:18.643335 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:19:18.643318    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623eae6f246e077e7046f3dfea75868df00ea060dab42e6b8bef108fbf99930c\": container with ID starting with 623eae6f246e077e7046f3dfea75868df00ea060dab42e6b8bef108fbf99930c not found: ID does not exist" containerID="623eae6f246e077e7046f3dfea75868df00ea060dab42e6b8bef108fbf99930c"
Jul 08 07:19:18.643389 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.643341    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623eae6f246e077e7046f3dfea75868df00ea060dab42e6b8bef108fbf99930c"} err="failed to get container status \"623eae6f246e077e7046f3dfea75868df00ea060dab42e6b8bef108fbf99930c\": rpc error: code = NotFound desc = could not find container \"623eae6f246e077e7046f3dfea75868df00ea060dab42e6b8bef108fbf99930c\": container with ID starting with 623eae6f246e077e7046f3dfea75868df00ea060dab42e6b8bef108fbf99930c not found: ID does not exist"
Jul 08 07:19:18.644517 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:18.644500    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-f84f3e-predictor-7bb4d4794-jfrtw"]
Jul 08 07:19:19.104595 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.104561    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"]
Jul 08 07:19:19.104889 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.104876    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kserve-container"
Jul 08 07:19:19.104940 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.104890    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kserve-container"
Jul 08 07:19:19.104940 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.104900    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="storage-initializer"
Jul 08 07:19:19.104940 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.104906    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="storage-initializer"
Jul 08 07:19:19.104940 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.104911    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kube-rbac-proxy"
Jul 08 07:19:19.104940 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.104916    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kube-rbac-proxy"
Jul 08 07:19:19.104940 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.104925    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2769bb4f-9abd-465e-a9d3-c68dbf3feabf" containerName="storage-initializer"
Jul 08 07:19:19.104940 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.104931    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2769bb4f-9abd-465e-a9d3-c68dbf3feabf" containerName="storage-initializer"
Jul 08 07:19:19.105147 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.104984    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2769bb4f-9abd-465e-a9d3-c68dbf3feabf" containerName="storage-initializer"
Jul 08 07:19:19.105147 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.104991    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kserve-container"
Jul 08 07:19:19.105147 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.104998    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2769bb4f-9abd-465e-a9d3-c68dbf3feabf" containerName="storage-initializer"
Jul 08 07:19:19.105147 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.105004    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1397740-8702-41ff-a059-a667771fde92" containerName="kube-rbac-proxy"
Jul 08 07:19:19.105147 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.105064    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2769bb4f-9abd-465e-a9d3-c68dbf3feabf" containerName="storage-initializer"
Jul 08 07:19:19.105147 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.105070    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2769bb4f-9abd-465e-a9d3-c68dbf3feabf" containerName="storage-initializer"
Jul 08 07:19:19.109403 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.109384    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.111833 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.111809    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 07:19:19.111935 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.111824    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 07:19:19.111935 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.111833    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-e84408\""
Jul 08 07:19:19.111935 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.111837    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-e84408-dockercfg-fh7lx\""
Jul 08 07:19:19.112106 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.111994    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e84408-predictor-serving-cert\""
Jul 08 07:19:19.112229 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.112214    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 07:19:19.112229 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.112223    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e84408-kube-rbac-proxy-sar-config\""
Jul 08 07:19:19.117671 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.117646    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"]
Jul 08 07:19:19.194635 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.194594    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2bb2\" (UniqueName: \"kubernetes.io/projected/d0d868cb-5446-43c1-ae77-91b85dff5a53-kube-api-access-q2bb2\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.194635 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.194638    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0d868cb-5446-43c1-ae77-91b85dff5a53-cabundle-cert\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.194888 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.194663    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0d868cb-5446-43c1-ae77-91b85dff5a53-kserve-provision-location\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.194888 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.194719    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0d868cb-5446-43c1-ae77-91b85dff5a53-proxy-tls\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.194888 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.194737    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0d868cb-5446-43c1-ae77-91b85dff5a53-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.295648 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.295623    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0d868cb-5446-43c1-ae77-91b85dff5a53-kserve-provision-location\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.296071 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.295666    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0d868cb-5446-43c1-ae77-91b85dff5a53-proxy-tls\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.296071 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.295689    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0d868cb-5446-43c1-ae77-91b85dff5a53-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.296071 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.295727    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2bb2\" (UniqueName: \"kubernetes.io/projected/d0d868cb-5446-43c1-ae77-91b85dff5a53-kube-api-access-q2bb2\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.296071 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.295742    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0d868cb-5446-43c1-ae77-91b85dff5a53-cabundle-cert\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.296241 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.296108    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0d868cb-5446-43c1-ae77-91b85dff5a53-kserve-provision-location\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.296400 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.296378    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0d868cb-5446-43c1-ae77-91b85dff5a53-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.296459 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.296441    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0d868cb-5446-43c1-ae77-91b85dff5a53-cabundle-cert\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.298243 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.298223    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0d868cb-5446-43c1-ae77-91b85dff5a53-proxy-tls\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.304440 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.304413    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2bb2\" (UniqueName: \"kubernetes.io/projected/d0d868cb-5446-43c1-ae77-91b85dff5a53-kube-api-access-q2bb2\") pod \"isvc-init-fail-e84408-predictor-969d565-fvmzm\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") " pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.420060 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.419999    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:19.550026 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.550002    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"]
Jul 08 07:19:19.552418 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:19:19.552385    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d868cb_5446_43c1_ae77_91b85dff5a53.slice/crio-f2a8e1b7a30c4fd976262d7a1df940ab7484554d134c87f0decee60d827445c7 WatchSource:0}: Error finding container f2a8e1b7a30c4fd976262d7a1df940ab7484554d134c87f0decee60d827445c7: Status 404 returned error can't find the container with id f2a8e1b7a30c4fd976262d7a1df940ab7484554d134c87f0decee60d827445c7
Jul 08 07:19:19.625059 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.625031    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm" event={"ID":"d0d868cb-5446-43c1-ae77-91b85dff5a53","Type":"ContainerStarted","Data":"fd1ec3c527d4e39f7b57ef309027ed02a0a309443242d3db8d9016aea3376790"}
Jul 08 07:19:19.625183 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.625110    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm" event={"ID":"d0d868cb-5446-43c1-ae77-91b85dff5a53","Type":"ContainerStarted","Data":"f2a8e1b7a30c4fd976262d7a1df940ab7484554d134c87f0decee60d827445c7"}
Jul 08 07:19:19.786623 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:19.786592    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1397740-8702-41ff-a059-a667771fde92" path="/var/lib/kubelet/pods/d1397740-8702-41ff-a059-a667771fde92/volumes"
Jul 08 07:19:24.650442 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:24.650365    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e84408-predictor-969d565-fvmzm_d0d868cb-5446-43c1-ae77-91b85dff5a53/storage-initializer/0.log"
Jul 08 07:19:24.650442 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:24.650406    2557 generic.go:356] "Generic (PLEG): container finished" podID="d0d868cb-5446-43c1-ae77-91b85dff5a53" containerID="fd1ec3c527d4e39f7b57ef309027ed02a0a309443242d3db8d9016aea3376790" exitCode=1
Jul 08 07:19:24.650865 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:24.650475    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm" event={"ID":"d0d868cb-5446-43c1-ae77-91b85dff5a53","Type":"ContainerDied","Data":"fd1ec3c527d4e39f7b57ef309027ed02a0a309443242d3db8d9016aea3376790"}
Jul 08 07:19:25.655949 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:25.655919    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e84408-predictor-969d565-fvmzm_d0d868cb-5446-43c1-ae77-91b85dff5a53/storage-initializer/0.log"
Jul 08 07:19:25.656312 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:25.655982    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm" event={"ID":"d0d868cb-5446-43c1-ae77-91b85dff5a53","Type":"ContainerStarted","Data":"e2eaf52fe4191162f4c4ea9fc993c371b8f65d8dc4b6e6bdc40b0f8163bd9a2a"}
Jul 08 07:19:27.744924 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:27.744895    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e84408-predictor-969d565-fvmzm_d0d868cb-5446-43c1-ae77-91b85dff5a53/storage-initializer/0.log"
Jul 08 07:19:27.754950 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:27.754928    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e84408-predictor-969d565-fvmzm_d0d868cb-5446-43c1-ae77-91b85dff5a53/storage-initializer/0.log"
Jul 08 07:19:29.118955 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:29.118926    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"]
Jul 08 07:19:29.119336 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:29.119139    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm" podUID="d0d868cb-5446-43c1-ae77-91b85dff5a53" containerName="storage-initializer" containerID="cri-o://e2eaf52fe4191162f4c4ea9fc993c371b8f65d8dc4b6e6bdc40b0f8163bd9a2a" gracePeriod=30
Jul 08 07:19:30.059970 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.059947    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e84408-predictor-969d565-fvmzm_d0d868cb-5446-43c1-ae77-91b85dff5a53/storage-initializer/1.log"
Jul 08 07:19:30.060304 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.060288    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e84408-predictor-969d565-fvmzm_d0d868cb-5446-43c1-ae77-91b85dff5a53/storage-initializer/0.log"
Jul 08 07:19:30.062477 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.062458    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:30.178297 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.178231    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0d868cb-5446-43c1-ae77-91b85dff5a53-proxy-tls\") pod \"d0d868cb-5446-43c1-ae77-91b85dff5a53\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") "
Jul 08 07:19:30.178297 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.178263    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0d868cb-5446-43c1-ae77-91b85dff5a53-kube-rbac-proxy-sar-config\") pod \"d0d868cb-5446-43c1-ae77-91b85dff5a53\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") "
Jul 08 07:19:30.178726 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.178304    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2bb2\" (UniqueName: \"kubernetes.io/projected/d0d868cb-5446-43c1-ae77-91b85dff5a53-kube-api-access-q2bb2\") pod \"d0d868cb-5446-43c1-ae77-91b85dff5a53\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") "
Jul 08 07:19:30.178726 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.178339    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0d868cb-5446-43c1-ae77-91b85dff5a53-cabundle-cert\") pod \"d0d868cb-5446-43c1-ae77-91b85dff5a53\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") "
Jul 08 07:19:30.178726 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.178373    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0d868cb-5446-43c1-ae77-91b85dff5a53-kserve-provision-location\") pod \"d0d868cb-5446-43c1-ae77-91b85dff5a53\" (UID: \"d0d868cb-5446-43c1-ae77-91b85dff5a53\") "
Jul 08 07:19:30.178726 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.178581    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d868cb-5446-43c1-ae77-91b85dff5a53-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0d868cb-5446-43c1-ae77-91b85dff5a53" (UID: "d0d868cb-5446-43c1-ae77-91b85dff5a53"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:19:30.178726 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.178686    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d868cb-5446-43c1-ae77-91b85dff5a53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0d868cb-5446-43c1-ae77-91b85dff5a53" (UID: "d0d868cb-5446-43c1-ae77-91b85dff5a53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:19:30.178726 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.178718    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d868cb-5446-43c1-ae77-91b85dff5a53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0d868cb-5446-43c1-ae77-91b85dff5a53" (UID: "d0d868cb-5446-43c1-ae77-91b85dff5a53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:19:30.180514 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.180492    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d868cb-5446-43c1-ae77-91b85dff5a53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0d868cb-5446-43c1-ae77-91b85dff5a53" (UID: "d0d868cb-5446-43c1-ae77-91b85dff5a53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:19:30.180558 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.180500    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d868cb-5446-43c1-ae77-91b85dff5a53-kube-api-access-q2bb2" (OuterVolumeSpecName: "kube-api-access-q2bb2") pod "d0d868cb-5446-43c1-ae77-91b85dff5a53" (UID: "d0d868cb-5446-43c1-ae77-91b85dff5a53"). InnerVolumeSpecName "kube-api-access-q2bb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:19:30.279614 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.279582    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0d868cb-5446-43c1-ae77-91b85dff5a53-kserve-provision-location\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:30.279614 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.279608    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0d868cb-5446-43c1-ae77-91b85dff5a53-proxy-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:30.279614 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.279618    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0d868cb-5446-43c1-ae77-91b85dff5a53-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:30.279860 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.279627    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2bb2\" (UniqueName: \"kubernetes.io/projected/d0d868cb-5446-43c1-ae77-91b85dff5a53-kube-api-access-q2bb2\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:30.279860 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.279636    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0d868cb-5446-43c1-ae77-91b85dff5a53-cabundle-cert\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:19:30.677273 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.677245    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e84408-predictor-969d565-fvmzm_d0d868cb-5446-43c1-ae77-91b85dff5a53/storage-initializer/1.log"
Jul 08 07:19:30.677626 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.677611    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e84408-predictor-969d565-fvmzm_d0d868cb-5446-43c1-ae77-91b85dff5a53/storage-initializer/0.log"
Jul 08 07:19:30.677689 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.677647    2557 generic.go:356] "Generic (PLEG): container finished" podID="d0d868cb-5446-43c1-ae77-91b85dff5a53" containerID="e2eaf52fe4191162f4c4ea9fc993c371b8f65d8dc4b6e6bdc40b0f8163bd9a2a" exitCode=1
Jul 08 07:19:30.677819 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.677687    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm" event={"ID":"d0d868cb-5446-43c1-ae77-91b85dff5a53","Type":"ContainerDied","Data":"e2eaf52fe4191162f4c4ea9fc993c371b8f65d8dc4b6e6bdc40b0f8163bd9a2a"}
Jul 08 07:19:30.677819 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.677700    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"
Jul 08 07:19:30.677819 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.677733    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm" event={"ID":"d0d868cb-5446-43c1-ae77-91b85dff5a53","Type":"ContainerDied","Data":"f2a8e1b7a30c4fd976262d7a1df940ab7484554d134c87f0decee60d827445c7"}
Jul 08 07:19:30.677819 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.677780    2557 scope.go:117] "RemoveContainer" containerID="e2eaf52fe4191162f4c4ea9fc993c371b8f65d8dc4b6e6bdc40b0f8163bd9a2a"
Jul 08 07:19:30.686741 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.686725    2557 scope.go:117] "RemoveContainer" containerID="fd1ec3c527d4e39f7b57ef309027ed02a0a309443242d3db8d9016aea3376790"
Jul 08 07:19:30.693834 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.693672    2557 scope.go:117] "RemoveContainer" containerID="e2eaf52fe4191162f4c4ea9fc993c371b8f65d8dc4b6e6bdc40b0f8163bd9a2a"
Jul 08 07:19:30.693946 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:19:30.693918    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2eaf52fe4191162f4c4ea9fc993c371b8f65d8dc4b6e6bdc40b0f8163bd9a2a\": container with ID starting with e2eaf52fe4191162f4c4ea9fc993c371b8f65d8dc4b6e6bdc40b0f8163bd9a2a not found: ID does not exist" containerID="e2eaf52fe4191162f4c4ea9fc993c371b8f65d8dc4b6e6bdc40b0f8163bd9a2a"
Jul 08 07:19:30.694000 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.693941    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2eaf52fe4191162f4c4ea9fc993c371b8f65d8dc4b6e6bdc40b0f8163bd9a2a"} err="failed to get container status \"e2eaf52fe4191162f4c4ea9fc993c371b8f65d8dc4b6e6bdc40b0f8163bd9a2a\": rpc error: code = NotFound desc = could not find container \"e2eaf52fe4191162f4c4ea9fc993c371b8f65d8dc4b6e6bdc40b0f8163bd9a2a\": container with ID starting with e2eaf52fe4191162f4c4ea9fc993c371b8f65d8dc4b6e6bdc40b0f8163bd9a2a not found: ID does not exist"
Jul 08 07:19:30.694000 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.693957    2557 scope.go:117] "RemoveContainer" containerID="fd1ec3c527d4e39f7b57ef309027ed02a0a309443242d3db8d9016aea3376790"
Jul 08 07:19:30.694184 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:19:30.694163    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd1ec3c527d4e39f7b57ef309027ed02a0a309443242d3db8d9016aea3376790\": container with ID starting with fd1ec3c527d4e39f7b57ef309027ed02a0a309443242d3db8d9016aea3376790 not found: ID does not exist" containerID="fd1ec3c527d4e39f7b57ef309027ed02a0a309443242d3db8d9016aea3376790"
Jul 08 07:19:30.694248 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.694192    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd1ec3c527d4e39f7b57ef309027ed02a0a309443242d3db8d9016aea3376790"} err="failed to get container status \"fd1ec3c527d4e39f7b57ef309027ed02a0a309443242d3db8d9016aea3376790\": rpc error: code = NotFound desc = could not find container \"fd1ec3c527d4e39f7b57ef309027ed02a0a309443242d3db8d9016aea3376790\": container with ID starting with fd1ec3c527d4e39f7b57ef309027ed02a0a309443242d3db8d9016aea3376790 not found: ID does not exist"
Jul 08 07:19:30.716507 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.716485    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"]
Jul 08 07:19:30.721063 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:30.721040    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e84408-predictor-969d565-fvmzm"]
Jul 08 07:19:31.784907 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:31.784876    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0d868cb-5446-43c1-ae77-91b85dff5a53" path="/var/lib/kubelet/pods/d0d868cb-5446-43c1-ae77-91b85dff5a53/volumes"
Jul 08 07:19:34.308417 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.308385    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"]
Jul 08 07:19:34.308821 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.308685    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d868cb-5446-43c1-ae77-91b85dff5a53" containerName="storage-initializer"
Jul 08 07:19:34.308821 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.308698    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d868cb-5446-43c1-ae77-91b85dff5a53" containerName="storage-initializer"
Jul 08 07:19:34.308821 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.308769    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d868cb-5446-43c1-ae77-91b85dff5a53" containerName="storage-initializer"
Jul 08 07:19:34.308930 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.308830    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d868cb-5446-43c1-ae77-91b85dff5a53" containerName="storage-initializer"
Jul 08 07:19:34.308930 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.308836    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d868cb-5446-43c1-ae77-91b85dff5a53" containerName="storage-initializer"
Jul 08 07:19:34.308930 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.308886    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d868cb-5446-43c1-ae77-91b85dff5a53" containerName="storage-initializer"
Jul 08 07:19:34.313220 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.313204    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.315810 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.315744    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-78661-predictor-serving-cert\""
Jul 08 07:19:34.315947 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.315816    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f4gs4\""
Jul 08 07:19:34.315947 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.315771    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-78661-kube-rbac-proxy-sar-config\""
Jul 08 07:19:34.315947 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.315863    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 07:19:34.315947 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.315751    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 07:19:34.315947 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.315906    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 07:19:34.316632 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.316614    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 07:19:34.320942 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.320919    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"]
Jul 08 07:19:34.409080 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.409056    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kserve-provision-location\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.409193 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.409086    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a1e506b-3a92-49c2-9d3b-a8c16638207e-cabundle-cert\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.409193 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.409119    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh4tb\" (UniqueName: \"kubernetes.io/projected/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kube-api-access-kh4tb\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.409193 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.409168    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.409310 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.409225    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a1e506b-3a92-49c2-9d3b-a8c16638207e-proxy-tls\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.509745 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.509717    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a1e506b-3a92-49c2-9d3b-a8c16638207e-cabundle-cert\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.509889 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.509782    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh4tb\" (UniqueName: \"kubernetes.io/projected/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kube-api-access-kh4tb\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.509889 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.509813    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.509889 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.509858    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a1e506b-3a92-49c2-9d3b-a8c16638207e-proxy-tls\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.509889 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.509879    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kserve-provision-location\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.510239 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.510222    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kserve-provision-location\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.510393 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.510374    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a1e506b-3a92-49c2-9d3b-a8c16638207e-cabundle-cert\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.510439 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.510417    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.512508 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.512488    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a1e506b-3a92-49c2-9d3b-a8c16638207e-proxy-tls\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.517647 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.517618    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh4tb\" (UniqueName: \"kubernetes.io/projected/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kube-api-access-kh4tb\") pod \"raw-sklearn-78661-predictor-bf8977fc5-htw5c\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") " pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.624101 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.624051    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:34.759676 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:34.756444    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"]
Jul 08 07:19:34.761441 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:19:34.761413    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1e506b_3a92_49c2_9d3b_a8c16638207e.slice/crio-6fe97c1afeeaa7879737bf47e54af0eaab7228448b901cca964d5250389303b8 WatchSource:0}: Error finding container 6fe97c1afeeaa7879737bf47e54af0eaab7228448b901cca964d5250389303b8: Status 404 returned error can't find the container with id 6fe97c1afeeaa7879737bf47e54af0eaab7228448b901cca964d5250389303b8
Jul 08 07:19:35.699914 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:35.699876    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" event={"ID":"9a1e506b-3a92-49c2-9d3b-a8c16638207e","Type":"ContainerStarted","Data":"49b3a68b56a500148ab2d37219831c7955f519732f9141bb923f3304ae623658"}
Jul 08 07:19:35.700290 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:35.699936    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" event={"ID":"9a1e506b-3a92-49c2-9d3b-a8c16638207e","Type":"ContainerStarted","Data":"6fe97c1afeeaa7879737bf47e54af0eaab7228448b901cca964d5250389303b8"}
Jul 08 07:19:38.712498 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:38.712419    2557 generic.go:356] "Generic (PLEG): container finished" podID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerID="49b3a68b56a500148ab2d37219831c7955f519732f9141bb923f3304ae623658" exitCode=0
Jul 08 07:19:38.712498 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:38.712484    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" event={"ID":"9a1e506b-3a92-49c2-9d3b-a8c16638207e","Type":"ContainerDied","Data":"49b3a68b56a500148ab2d37219831c7955f519732f9141bb923f3304ae623658"}
Jul 08 07:19:39.719120 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:39.719086    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" event={"ID":"9a1e506b-3a92-49c2-9d3b-a8c16638207e","Type":"ContainerStarted","Data":"2535260ef77760c32796b4471b848c3d66272f44bbf3cf6a32d8dd6260eb981f"}
Jul 08 07:19:39.719494 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:39.719147    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" event={"ID":"9a1e506b-3a92-49c2-9d3b-a8c16638207e","Type":"ContainerStarted","Data":"43ecb9a7eebac5eb5b43e15573a658c567952a1677be89b655cbf3c165d9a44c"}
Jul 08 07:19:39.719494 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:39.719326    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:39.737698 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:39.737648    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" podStartSLOduration=2.196661741 podStartE2EDuration="5.737635821s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.54097408s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:19:39.736134391 +0000 UTC m=+1512.472479105" watchObservedRunningTime="2026-07-08 07:19:39.737635821 +0000 UTC m=+1512.473980535"
Jul 08 07:19:40.723598 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:40.723563    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:40.724837 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:40.724812    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 07:19:41.727578 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:41.727536    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 07:19:46.732872 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:46.732847    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:19:46.733342 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:46.733312    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 07:19:56.733964 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:19:56.733920    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 07:20:06.733297 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:20:06.733259    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 07:20:16.733799 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:20:16.733738    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 07:20:26.733843 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:20:26.733802    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 07:20:36.733555 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:20:36.733518    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 07:20:46.733865 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:20:46.733823    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 07:20:56.734738 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:20:56.734704    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:21:04.454921 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:04.454889    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"]
Jul 08 07:21:04.455469 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:04.455193    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kserve-container" containerID="cri-o://43ecb9a7eebac5eb5b43e15573a658c567952a1677be89b655cbf3c165d9a44c" gracePeriod=30
Jul 08 07:21:04.455469 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:04.455336    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kube-rbac-proxy" containerID="cri-o://2535260ef77760c32796b4471b848c3d66272f44bbf3cf6a32d8dd6260eb981f" gracePeriod=30
Jul 08 07:21:05.057960 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:05.057929    2557 generic.go:356] "Generic (PLEG): container finished" podID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerID="2535260ef77760c32796b4471b848c3d66272f44bbf3cf6a32d8dd6260eb981f" exitCode=2
Jul 08 07:21:05.058126 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:05.057977    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" event={"ID":"9a1e506b-3a92-49c2-9d3b-a8c16638207e","Type":"ContainerDied","Data":"2535260ef77760c32796b4471b848c3d66272f44bbf3cf6a32d8dd6260eb981f"}
Jul 08 07:21:06.728421 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:06.728380    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 08 07:21:06.733829 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:06.733803    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 08 07:21:08.702514 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.702490    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:21:08.729914 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.729894    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a1e506b-3a92-49c2-9d3b-a8c16638207e-proxy-tls\") pod \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") "
Jul 08 07:21:08.730041 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.729949    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh4tb\" (UniqueName: \"kubernetes.io/projected/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kube-api-access-kh4tb\") pod \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") "
Jul 08 07:21:08.730041 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.729991    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kserve-provision-location\") pod \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") "
Jul 08 07:21:08.730041 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.730019    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a1e506b-3a92-49c2-9d3b-a8c16638207e-cabundle-cert\") pod \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") "
Jul 08 07:21:08.730224 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.730041    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kube-rbac-proxy-sar-config\") pod \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\" (UID: \"9a1e506b-3a92-49c2-9d3b-a8c16638207e\") "
Jul 08 07:21:08.730388 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.730354    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9a1e506b-3a92-49c2-9d3b-a8c16638207e" (UID: "9a1e506b-3a92-49c2-9d3b-a8c16638207e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:21:08.730457 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.730378    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1e506b-3a92-49c2-9d3b-a8c16638207e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9a1e506b-3a92-49c2-9d3b-a8c16638207e" (UID: "9a1e506b-3a92-49c2-9d3b-a8c16638207e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:21:08.730457 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.730434    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9a1e506b-3a92-49c2-9d3b-a8c16638207e" (UID: "9a1e506b-3a92-49c2-9d3b-a8c16638207e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:21:08.732076 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.732056    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kube-api-access-kh4tb" (OuterVolumeSpecName: "kube-api-access-kh4tb") pod "9a1e506b-3a92-49c2-9d3b-a8c16638207e" (UID: "9a1e506b-3a92-49c2-9d3b-a8c16638207e"). InnerVolumeSpecName "kube-api-access-kh4tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:21:08.732224 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.732206    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1e506b-3a92-49c2-9d3b-a8c16638207e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9a1e506b-3a92-49c2-9d3b-a8c16638207e" (UID: "9a1e506b-3a92-49c2-9d3b-a8c16638207e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:21:08.831085 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.831057    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kserve-provision-location\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:21:08.831085 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.831079    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a1e506b-3a92-49c2-9d3b-a8c16638207e-cabundle-cert\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:21:08.831085 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.831089    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:21:08.831263 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.831100    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a1e506b-3a92-49c2-9d3b-a8c16638207e-proxy-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:21:08.831263 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:08.831108    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh4tb\" (UniqueName: \"kubernetes.io/projected/9a1e506b-3a92-49c2-9d3b-a8c16638207e-kube-api-access-kh4tb\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:21:09.075431 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.075351    2557 generic.go:356] "Generic (PLEG): container finished" podID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerID="43ecb9a7eebac5eb5b43e15573a658c567952a1677be89b655cbf3c165d9a44c" exitCode=0
Jul 08 07:21:09.075431 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.075408    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"
Jul 08 07:21:09.075578 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.075441    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" event={"ID":"9a1e506b-3a92-49c2-9d3b-a8c16638207e","Type":"ContainerDied","Data":"43ecb9a7eebac5eb5b43e15573a658c567952a1677be89b655cbf3c165d9a44c"}
Jul 08 07:21:09.075578 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.075503    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c" event={"ID":"9a1e506b-3a92-49c2-9d3b-a8c16638207e","Type":"ContainerDied","Data":"6fe97c1afeeaa7879737bf47e54af0eaab7228448b901cca964d5250389303b8"}
Jul 08 07:21:09.075578 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.075530    2557 scope.go:117] "RemoveContainer" containerID="2535260ef77760c32796b4471b848c3d66272f44bbf3cf6a32d8dd6260eb981f"
Jul 08 07:21:09.086301 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.086085    2557 scope.go:117] "RemoveContainer" containerID="43ecb9a7eebac5eb5b43e15573a658c567952a1677be89b655cbf3c165d9a44c"
Jul 08 07:21:09.093505 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.093479    2557 scope.go:117] "RemoveContainer" containerID="49b3a68b56a500148ab2d37219831c7955f519732f9141bb923f3304ae623658"
Jul 08 07:21:09.097450 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.097430    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"]
Jul 08 07:21:09.101226 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.101133    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-78661-predictor-bf8977fc5-htw5c"]
Jul 08 07:21:09.101292 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.101235    2557 scope.go:117] "RemoveContainer" containerID="2535260ef77760c32796b4471b848c3d66272f44bbf3cf6a32d8dd6260eb981f"
Jul 08 07:21:09.101504 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:21:09.101484    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2535260ef77760c32796b4471b848c3d66272f44bbf3cf6a32d8dd6260eb981f\": container with ID starting with 2535260ef77760c32796b4471b848c3d66272f44bbf3cf6a32d8dd6260eb981f not found: ID does not exist" containerID="2535260ef77760c32796b4471b848c3d66272f44bbf3cf6a32d8dd6260eb981f"
Jul 08 07:21:09.101571 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.101514    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2535260ef77760c32796b4471b848c3d66272f44bbf3cf6a32d8dd6260eb981f"} err="failed to get container status \"2535260ef77760c32796b4471b848c3d66272f44bbf3cf6a32d8dd6260eb981f\": rpc error: code = NotFound desc = could not find container \"2535260ef77760c32796b4471b848c3d66272f44bbf3cf6a32d8dd6260eb981f\": container with ID starting with 2535260ef77760c32796b4471b848c3d66272f44bbf3cf6a32d8dd6260eb981f not found: ID does not exist"
Jul 08 07:21:09.101571 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.101537    2557 scope.go:117] "RemoveContainer" containerID="43ecb9a7eebac5eb5b43e15573a658c567952a1677be89b655cbf3c165d9a44c"
Jul 08 07:21:09.101769 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:21:09.101736    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ecb9a7eebac5eb5b43e15573a658c567952a1677be89b655cbf3c165d9a44c\": container with ID starting with 43ecb9a7eebac5eb5b43e15573a658c567952a1677be89b655cbf3c165d9a44c not found: ID does not exist" containerID="43ecb9a7eebac5eb5b43e15573a658c567952a1677be89b655cbf3c165d9a44c"
Jul 08 07:21:09.101843 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.101785    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ecb9a7eebac5eb5b43e15573a658c567952a1677be89b655cbf3c165d9a44c"} err="failed to get container status \"43ecb9a7eebac5eb5b43e15573a658c567952a1677be89b655cbf3c165d9a44c\": rpc error: code = NotFound desc = could not find container \"43ecb9a7eebac5eb5b43e15573a658c567952a1677be89b655cbf3c165d9a44c\": container with ID starting with 43ecb9a7eebac5eb5b43e15573a658c567952a1677be89b655cbf3c165d9a44c not found: ID does not exist"
Jul 08 07:21:09.101843 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.101806    2557 scope.go:117] "RemoveContainer" containerID="49b3a68b56a500148ab2d37219831c7955f519732f9141bb923f3304ae623658"
Jul 08 07:21:09.102031 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:21:09.102016    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b3a68b56a500148ab2d37219831c7955f519732f9141bb923f3304ae623658\": container with ID starting with 49b3a68b56a500148ab2d37219831c7955f519732f9141bb923f3304ae623658 not found: ID does not exist" containerID="49b3a68b56a500148ab2d37219831c7955f519732f9141bb923f3304ae623658"
Jul 08 07:21:09.102080 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.102037    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b3a68b56a500148ab2d37219831c7955f519732f9141bb923f3304ae623658"} err="failed to get container status \"49b3a68b56a500148ab2d37219831c7955f519732f9141bb923f3304ae623658\": rpc error: code = NotFound desc = could not find container \"49b3a68b56a500148ab2d37219831c7955f519732f9141bb923f3304ae623658\": container with ID starting with 49b3a68b56a500148ab2d37219831c7955f519732f9141bb923f3304ae623658 not found: ID does not exist"
Jul 08 07:21:09.610016 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.609984    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"]
Jul 08 07:21:09.610323 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.610311    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kserve-container"
Jul 08 07:21:09.610373 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.610326    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kserve-container"
Jul 08 07:21:09.610373 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.610338    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="storage-initializer"
Jul 08 07:21:09.610373 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.610344    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="storage-initializer"
Jul 08 07:21:09.610373 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.610357    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kube-rbac-proxy"
Jul 08 07:21:09.610373 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.610362    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kube-rbac-proxy"
Jul 08 07:21:09.610518 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.610412    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kserve-container"
Jul 08 07:21:09.610518 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.610422    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" containerName="kube-rbac-proxy"
Jul 08 07:21:09.614744 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.614727    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.617505 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.617485    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-24704-kube-rbac-proxy-sar-config\""
Jul 08 07:21:09.617708 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.617538    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 07:21:09.617708 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.617570    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 07:21:09.617708 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.617585    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-24704-predictor-serving-cert\""
Jul 08 07:21:09.617708 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.617570    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-f4gs4\""
Jul 08 07:21:09.618235 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.618220    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 07:21:09.618282 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.618250    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 07:21:09.623339 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.623317    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"]
Jul 08 07:21:09.642177 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.642133    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.642333 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.642192    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2ac5022-59fc-4636-9d09-dc8e7ec76662-cabundle-cert\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.642333 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.642231    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kserve-provision-location\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.642333 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.642302    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2ac5022-59fc-4636-9d09-dc8e7ec76662-proxy-tls\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.642333 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.642328    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nq6p\" (UniqueName: \"kubernetes.io/projected/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kube-api-access-4nq6p\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.742669 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.742639    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.742669 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.742671    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2ac5022-59fc-4636-9d09-dc8e7ec76662-cabundle-cert\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.743134 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.742692    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kserve-provision-location\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.743134 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.742724    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2ac5022-59fc-4636-9d09-dc8e7ec76662-proxy-tls\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.743134 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.742839    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nq6p\" (UniqueName: \"kubernetes.io/projected/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kube-api-access-4nq6p\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.743134 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:21:09.742848    2557 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-serving-cert: secret "raw-sklearn-runtime-24704-predictor-serving-cert" not found
Jul 08 07:21:09.743134 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:21:09.742948    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2ac5022-59fc-4636-9d09-dc8e7ec76662-proxy-tls podName:e2ac5022-59fc-4636-9d09-dc8e7ec76662 nodeName:}" failed. No retries permitted until 2026-07-08 07:21:10.242926591 +0000 UTC m=+1602.979271283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e2ac5022-59fc-4636-9d09-dc8e7ec76662-proxy-tls") pod "raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" (UID: "e2ac5022-59fc-4636-9d09-dc8e7ec76662") : secret "raw-sklearn-runtime-24704-predictor-serving-cert" not found
Jul 08 07:21:09.743347 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.743175    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kserve-provision-location\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.743347 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.743314    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2ac5022-59fc-4636-9d09-dc8e7ec76662-cabundle-cert\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.743347 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.743340    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.751729 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.751706    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nq6p\" (UniqueName: \"kubernetes.io/projected/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kube-api-access-4nq6p\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:09.785324 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:09.785298    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a1e506b-3a92-49c2-9d3b-a8c16638207e" path="/var/lib/kubelet/pods/9a1e506b-3a92-49c2-9d3b-a8c16638207e/volumes"
Jul 08 07:21:10.246128 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:10.246094    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2ac5022-59fc-4636-9d09-dc8e7ec76662-proxy-tls\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:10.248661 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:10.248636    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2ac5022-59fc-4636-9d09-dc8e7ec76662-proxy-tls\") pod \"raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:10.525668 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:10.525587    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:10.655534 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:10.655509    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"]
Jul 08 07:21:10.656927 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:21:10.656893    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ac5022_59fc_4636_9d09_dc8e7ec76662.slice/crio-2581e419e6ddf07e6cd74300198f3641ce6101c6dbbd4ba37dc9a437930491f0 WatchSource:0}: Error finding container 2581e419e6ddf07e6cd74300198f3641ce6101c6dbbd4ba37dc9a437930491f0: Status 404 returned error can't find the container with id 2581e419e6ddf07e6cd74300198f3641ce6101c6dbbd4ba37dc9a437930491f0
Jul 08 07:21:11.085695 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:11.085648    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" event={"ID":"e2ac5022-59fc-4636-9d09-dc8e7ec76662","Type":"ContainerStarted","Data":"506ca0d2088d74abf546448e3aaa1b4d6e037cc88d4f954c8f15b19906af98bc"}
Jul 08 07:21:11.086097 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:11.085726    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" event={"ID":"e2ac5022-59fc-4636-9d09-dc8e7ec76662","Type":"ContainerStarted","Data":"2581e419e6ddf07e6cd74300198f3641ce6101c6dbbd4ba37dc9a437930491f0"}
Jul 08 07:21:15.101996 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:15.101964    2557 generic.go:356] "Generic (PLEG): container finished" podID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerID="506ca0d2088d74abf546448e3aaa1b4d6e037cc88d4f954c8f15b19906af98bc" exitCode=0
Jul 08 07:21:15.102388 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:15.102040    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" event={"ID":"e2ac5022-59fc-4636-9d09-dc8e7ec76662","Type":"ContainerDied","Data":"506ca0d2088d74abf546448e3aaa1b4d6e037cc88d4f954c8f15b19906af98bc"}
Jul 08 07:21:16.108825 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:16.108790    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" event={"ID":"e2ac5022-59fc-4636-9d09-dc8e7ec76662","Type":"ContainerStarted","Data":"a4e0345cb6034b8ab7f7b62327057331fe58f39db13dbf951d5555165ab02125"}
Jul 08 07:21:16.109232 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:16.108849    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" event={"ID":"e2ac5022-59fc-4636-9d09-dc8e7ec76662","Type":"ContainerStarted","Data":"518074ae49dd202ad835a299adb4a01fc590e7be509606c15c4adb2ff8207d3f"}
Jul 08 07:21:16.109232 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:16.109031    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:16.129375 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:16.129330    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" podStartSLOduration=3.196735575 podStartE2EDuration="7.129315962s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.932580387s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:21:16.12837994 +0000 UTC m=+1608.864724655" watchObservedRunningTime="2026-07-08 07:21:16.129315962 +0000 UTC m=+1608.865660676"
Jul 08 07:21:17.113706 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:17.113670    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:17.114880 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:17.114853    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 07:21:18.117287 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:18.117254    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 07:21:23.122055 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:23.121972    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:21:23.122477 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:23.122450    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 07:21:33.122453 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:33.122408    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 07:21:43.123079 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:43.123037    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 07:21:53.122970 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:21:53.122925    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 07:22:03.122432 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:03.122389    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 07:22:13.122455 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:13.122416    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 07:22:23.122987 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:23.122951    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 07:22:33.123661 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:33.123630    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:22:39.709600 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:39.709568    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"]
Jul 08 07:22:39.710038 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:39.709809    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kserve-container" containerID="cri-o://518074ae49dd202ad835a299adb4a01fc590e7be509606c15c4adb2ff8207d3f" gracePeriod=30
Jul 08 07:22:39.710038 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:39.709904    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kube-rbac-proxy" containerID="cri-o://a4e0345cb6034b8ab7f7b62327057331fe58f39db13dbf951d5555165ab02125" gracePeriod=30
Jul 08 07:22:40.448006 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:40.447974    2557 generic.go:356] "Generic (PLEG): container finished" podID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerID="a4e0345cb6034b8ab7f7b62327057331fe58f39db13dbf951d5555165ab02125" exitCode=2
Jul 08 07:22:40.448184 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:40.448047    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" event={"ID":"e2ac5022-59fc-4636-9d09-dc8e7ec76662","Type":"ContainerDied","Data":"a4e0345cb6034b8ab7f7b62327057331fe58f39db13dbf951d5555165ab02125"}
Jul 08 07:22:43.118310 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:43.118267    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 08 07:22:43.122630 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:43.122603    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 08 07:22:44.149971 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.149947    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:22:44.244929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.244870    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2ac5022-59fc-4636-9d09-dc8e7ec76662-cabundle-cert\") pod \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") "
Jul 08 07:22:44.244929 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.244911    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nq6p\" (UniqueName: \"kubernetes.io/projected/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kube-api-access-4nq6p\") pod \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") "
Jul 08 07:22:44.245094 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.244940    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kserve-provision-location\") pod \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") "
Jul 08 07:22:44.245094 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.244969    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kube-rbac-proxy-sar-config\") pod \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") "
Jul 08 07:22:44.245094 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.244999    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2ac5022-59fc-4636-9d09-dc8e7ec76662-proxy-tls\") pod \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\" (UID: \"e2ac5022-59fc-4636-9d09-dc8e7ec76662\") "
Jul 08 07:22:44.245270 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.245245    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ac5022-59fc-4636-9d09-dc8e7ec76662-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e2ac5022-59fc-4636-9d09-dc8e7ec76662" (UID: "e2ac5022-59fc-4636-9d09-dc8e7ec76662"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:22:44.245332 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.245282    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2ac5022-59fc-4636-9d09-dc8e7ec76662" (UID: "e2ac5022-59fc-4636-9d09-dc8e7ec76662"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:22:44.245332 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.245309    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e2ac5022-59fc-4636-9d09-dc8e7ec76662" (UID: "e2ac5022-59fc-4636-9d09-dc8e7ec76662"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 07:22:44.247050 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.247029    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kube-api-access-4nq6p" (OuterVolumeSpecName: "kube-api-access-4nq6p") pod "e2ac5022-59fc-4636-9d09-dc8e7ec76662" (UID: "e2ac5022-59fc-4636-9d09-dc8e7ec76662"). InnerVolumeSpecName "kube-api-access-4nq6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:22:44.247293 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.247273    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2ac5022-59fc-4636-9d09-dc8e7ec76662-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e2ac5022-59fc-4636-9d09-dc8e7ec76662" (UID: "e2ac5022-59fc-4636-9d09-dc8e7ec76662"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 07:22:44.345894 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.345862    2557 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e2ac5022-59fc-4636-9d09-dc8e7ec76662-proxy-tls\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:22:44.345894 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.345889    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e2ac5022-59fc-4636-9d09-dc8e7ec76662-cabundle-cert\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:22:44.346076 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.345903    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nq6p\" (UniqueName: \"kubernetes.io/projected/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kube-api-access-4nq6p\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:22:44.346076 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.345916    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kserve-provision-location\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:22:44.346076 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.345928    2557 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e2ac5022-59fc-4636-9d09-dc8e7ec76662-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:22:44.466592 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.466555    2557 generic.go:356] "Generic (PLEG): container finished" podID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerID="518074ae49dd202ad835a299adb4a01fc590e7be509606c15c4adb2ff8207d3f" exitCode=0
Jul 08 07:22:44.466747 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.466614    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"
Jul 08 07:22:44.466747 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.466638    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" event={"ID":"e2ac5022-59fc-4636-9d09-dc8e7ec76662","Type":"ContainerDied","Data":"518074ae49dd202ad835a299adb4a01fc590e7be509606c15c4adb2ff8207d3f"}
Jul 08 07:22:44.466747 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.466696    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b" event={"ID":"e2ac5022-59fc-4636-9d09-dc8e7ec76662","Type":"ContainerDied","Data":"2581e419e6ddf07e6cd74300198f3641ce6101c6dbbd4ba37dc9a437930491f0"}
Jul 08 07:22:44.466747 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.466723    2557 scope.go:117] "RemoveContainer" containerID="a4e0345cb6034b8ab7f7b62327057331fe58f39db13dbf951d5555165ab02125"
Jul 08 07:22:44.475534 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.475521    2557 scope.go:117] "RemoveContainer" containerID="518074ae49dd202ad835a299adb4a01fc590e7be509606c15c4adb2ff8207d3f"
Jul 08 07:22:44.482808 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.482788    2557 scope.go:117] "RemoveContainer" containerID="506ca0d2088d74abf546448e3aaa1b4d6e037cc88d4f954c8f15b19906af98bc"
Jul 08 07:22:44.489738 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.489721    2557 scope.go:117] "RemoveContainer" containerID="a4e0345cb6034b8ab7f7b62327057331fe58f39db13dbf951d5555165ab02125"
Jul 08 07:22:44.490036 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:22:44.490002    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e0345cb6034b8ab7f7b62327057331fe58f39db13dbf951d5555165ab02125\": container with ID starting with a4e0345cb6034b8ab7f7b62327057331fe58f39db13dbf951d5555165ab02125 not found: ID does not exist" containerID="a4e0345cb6034b8ab7f7b62327057331fe58f39db13dbf951d5555165ab02125"
Jul 08 07:22:44.490184 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.490046    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e0345cb6034b8ab7f7b62327057331fe58f39db13dbf951d5555165ab02125"} err="failed to get container status \"a4e0345cb6034b8ab7f7b62327057331fe58f39db13dbf951d5555165ab02125\": rpc error: code = NotFound desc = could not find container \"a4e0345cb6034b8ab7f7b62327057331fe58f39db13dbf951d5555165ab02125\": container with ID starting with a4e0345cb6034b8ab7f7b62327057331fe58f39db13dbf951d5555165ab02125 not found: ID does not exist"
Jul 08 07:22:44.490184 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.490069    2557 scope.go:117] "RemoveContainer" containerID="518074ae49dd202ad835a299adb4a01fc590e7be509606c15c4adb2ff8207d3f"
Jul 08 07:22:44.490537 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:22:44.490407    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"518074ae49dd202ad835a299adb4a01fc590e7be509606c15c4adb2ff8207d3f\": container with ID starting with 518074ae49dd202ad835a299adb4a01fc590e7be509606c15c4adb2ff8207d3f not found: ID does not exist" containerID="518074ae49dd202ad835a299adb4a01fc590e7be509606c15c4adb2ff8207d3f"
Jul 08 07:22:44.490537 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.490439    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"518074ae49dd202ad835a299adb4a01fc590e7be509606c15c4adb2ff8207d3f"} err="failed to get container status \"518074ae49dd202ad835a299adb4a01fc590e7be509606c15c4adb2ff8207d3f\": rpc error: code = NotFound desc = could not find container \"518074ae49dd202ad835a299adb4a01fc590e7be509606c15c4adb2ff8207d3f\": container with ID starting with 518074ae49dd202ad835a299adb4a01fc590e7be509606c15c4adb2ff8207d3f not found: ID does not exist"
Jul 08 07:22:44.490537 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.490471    2557 scope.go:117] "RemoveContainer" containerID="506ca0d2088d74abf546448e3aaa1b4d6e037cc88d4f954c8f15b19906af98bc"
Jul 08 07:22:44.491045 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:22:44.491020    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506ca0d2088d74abf546448e3aaa1b4d6e037cc88d4f954c8f15b19906af98bc\": container with ID starting with 506ca0d2088d74abf546448e3aaa1b4d6e037cc88d4f954c8f15b19906af98bc not found: ID does not exist" containerID="506ca0d2088d74abf546448e3aaa1b4d6e037cc88d4f954c8f15b19906af98bc"
Jul 08 07:22:44.491148 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.491048    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506ca0d2088d74abf546448e3aaa1b4d6e037cc88d4f954c8f15b19906af98bc"} err="failed to get container status \"506ca0d2088d74abf546448e3aaa1b4d6e037cc88d4f954c8f15b19906af98bc\": rpc error: code = NotFound desc = could not find container \"506ca0d2088d74abf546448e3aaa1b4d6e037cc88d4f954c8f15b19906af98bc\": container with ID starting with 506ca0d2088d74abf546448e3aaa1b4d6e037cc88d4f954c8f15b19906af98bc not found: ID does not exist"
Jul 08 07:22:44.492398 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.492377    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"]
Jul 08 07:22:44.496630 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:44.496580    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-24704-predictor-656b49b9c5-xf97b"]
Jul 08 07:22:45.784625 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.784595    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" path="/var/lib/kubelet/pods/e2ac5022-59fc-4636-9d09-dc8e7ec76662/volumes"
Jul 08 07:22:45.835581 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.835550    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2mqp7/must-gather-6qb6p"]
Jul 08 07:22:45.835856 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.835844    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kserve-container"
Jul 08 07:22:45.835906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.835858    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kserve-container"
Jul 08 07:22:45.835906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.835868    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="storage-initializer"
Jul 08 07:22:45.835906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.835873    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="storage-initializer"
Jul 08 07:22:45.835906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.835895    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kube-rbac-proxy"
Jul 08 07:22:45.835906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.835900    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kube-rbac-proxy"
Jul 08 07:22:45.836056 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.835948    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kserve-container"
Jul 08 07:22:45.836056 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.835958    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ac5022-59fc-4636-9d09-dc8e7ec76662" containerName="kube-rbac-proxy"
Jul 08 07:22:45.840361 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.840345    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2mqp7/must-gather-6qb6p"
Jul 08 07:22:45.843211 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.843187    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2mqp7\"/\"openshift-service-ca.crt\""
Jul 08 07:22:45.843345 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.843214    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2mqp7\"/\"default-dockercfg-hgx8s\""
Jul 08 07:22:45.843345 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.843243    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2mqp7\"/\"kube-root-ca.crt\""
Jul 08 07:22:45.850628 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.850606    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2mqp7/must-gather-6qb6p"]
Jul 08 07:22:45.957443 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.957415    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxsm7\" (UniqueName: \"kubernetes.io/projected/652b1071-2ed0-4e41-8095-dcb11d212b99-kube-api-access-jxsm7\") pod \"must-gather-6qb6p\" (UID: \"652b1071-2ed0-4e41-8095-dcb11d212b99\") " pod="openshift-must-gather-2mqp7/must-gather-6qb6p"
Jul 08 07:22:45.957591 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:45.957459    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/652b1071-2ed0-4e41-8095-dcb11d212b99-must-gather-output\") pod \"must-gather-6qb6p\" (UID: \"652b1071-2ed0-4e41-8095-dcb11d212b99\") " pod="openshift-must-gather-2mqp7/must-gather-6qb6p"
Jul 08 07:22:46.058742 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:46.058680    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxsm7\" (UniqueName: \"kubernetes.io/projected/652b1071-2ed0-4e41-8095-dcb11d212b99-kube-api-access-jxsm7\") pod \"must-gather-6qb6p\" (UID: \"652b1071-2ed0-4e41-8095-dcb11d212b99\") " pod="openshift-must-gather-2mqp7/must-gather-6qb6p"
Jul 08 07:22:46.058742 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:46.058726    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/652b1071-2ed0-4e41-8095-dcb11d212b99-must-gather-output\") pod \"must-gather-6qb6p\" (UID: \"652b1071-2ed0-4e41-8095-dcb11d212b99\") " pod="openshift-must-gather-2mqp7/must-gather-6qb6p"
Jul 08 07:22:46.059048 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:46.059033    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/652b1071-2ed0-4e41-8095-dcb11d212b99-must-gather-output\") pod \"must-gather-6qb6p\" (UID: \"652b1071-2ed0-4e41-8095-dcb11d212b99\") " pod="openshift-must-gather-2mqp7/must-gather-6qb6p"
Jul 08 07:22:46.067066 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:46.067043    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxsm7\" (UniqueName: \"kubernetes.io/projected/652b1071-2ed0-4e41-8095-dcb11d212b99-kube-api-access-jxsm7\") pod \"must-gather-6qb6p\" (UID: \"652b1071-2ed0-4e41-8095-dcb11d212b99\") " pod="openshift-must-gather-2mqp7/must-gather-6qb6p"
Jul 08 07:22:46.158876 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:46.158841    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2mqp7/must-gather-6qb6p"
Jul 08 07:22:46.286966 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:46.286934    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2mqp7/must-gather-6qb6p"]
Jul 08 07:22:46.289779 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:22:46.289734    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod652b1071_2ed0_4e41_8095_dcb11d212b99.slice/crio-0b4fe9fac8eba5b53a7ec1bf503fa1dec6abf910628944425e88e9bcdc76eafb WatchSource:0}: Error finding container 0b4fe9fac8eba5b53a7ec1bf503fa1dec6abf910628944425e88e9bcdc76eafb: Status 404 returned error can't find the container with id 0b4fe9fac8eba5b53a7ec1bf503fa1dec6abf910628944425e88e9bcdc76eafb
Jul 08 07:22:46.291751 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:46.291734    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 07:22:46.476182 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:46.476146    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2mqp7/must-gather-6qb6p" event={"ID":"652b1071-2ed0-4e41-8095-dcb11d212b99","Type":"ContainerStarted","Data":"0b4fe9fac8eba5b53a7ec1bf503fa1dec6abf910628944425e88e9bcdc76eafb"}
Jul 08 07:22:53.511030 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:53.510987    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2mqp7/must-gather-6qb6p" event={"ID":"652b1071-2ed0-4e41-8095-dcb11d212b99","Type":"ContainerStarted","Data":"63048b640015c74f5e849fef2de08fc417a234ca6a644e1a7395bd6683fc7dfa"}
Jul 08 07:22:53.511382 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:53.511044    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2mqp7/must-gather-6qb6p" event={"ID":"652b1071-2ed0-4e41-8095-dcb11d212b99","Type":"ContainerStarted","Data":"2d6a95d96addd08ca72aee90f016f5483075636a2d3028e8cef20f7da9b451e6"}
Jul 08 07:22:53.527128 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:22:53.527081    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2mqp7/must-gather-6qb6p" podStartSLOduration=2.304309162 podStartE2EDuration="8.527067631s" totalImagesPullingTime="6.222758469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:22:53.525876911 +0000 UTC m=+1706.262221626" watchObservedRunningTime="2026-07-08 07:22:53.527067631 +0000 UTC m=+1706.263412347"
Jul 08 07:23:13.598320 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:13.598282    2557 generic.go:356] "Generic (PLEG): container finished" podID="652b1071-2ed0-4e41-8095-dcb11d212b99" containerID="2d6a95d96addd08ca72aee90f016f5483075636a2d3028e8cef20f7da9b451e6" exitCode=0
Jul 08 07:23:13.598850 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:13.598356    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2mqp7/must-gather-6qb6p" event={"ID":"652b1071-2ed0-4e41-8095-dcb11d212b99","Type":"ContainerDied","Data":"2d6a95d96addd08ca72aee90f016f5483075636a2d3028e8cef20f7da9b451e6"}
Jul 08 07:23:13.598850 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:13.598802    2557 scope.go:117] "RemoveContainer" containerID="2d6a95d96addd08ca72aee90f016f5483075636a2d3028e8cef20f7da9b451e6"
Jul 08 07:23:13.881559 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:13.881480    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2mqp7_must-gather-6qb6p_652b1071-2ed0-4e41-8095-dcb11d212b99/gather/0.log"
Jul 08 07:23:14.543111 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:14.543083    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jdzkf/must-gather-9l8mf"]
Jul 08 07:23:14.545582 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:14.545567    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jdzkf/must-gather-9l8mf"
Jul 08 07:23:14.548254 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:14.548231    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jdzkf\"/\"default-dockercfg-97tdw\""
Jul 08 07:23:14.548375 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:14.548235    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jdzkf\"/\"openshift-service-ca.crt\""
Jul 08 07:23:14.548375 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:14.548268    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jdzkf\"/\"kube-root-ca.crt\""
Jul 08 07:23:14.557843 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:14.557818    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jdzkf/must-gather-9l8mf"]
Jul 08 07:23:14.707679 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:14.707646    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74t4g\" (UniqueName: \"kubernetes.io/projected/84b74e7a-6168-4e5a-ad50-981250b6bea6-kube-api-access-74t4g\") pod \"must-gather-9l8mf\" (UID: \"84b74e7a-6168-4e5a-ad50-981250b6bea6\") " pod="openshift-must-gather-jdzkf/must-gather-9l8mf"
Jul 08 07:23:14.708044 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:14.707698    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/84b74e7a-6168-4e5a-ad50-981250b6bea6-must-gather-output\") pod \"must-gather-9l8mf\" (UID: \"84b74e7a-6168-4e5a-ad50-981250b6bea6\") " pod="openshift-must-gather-jdzkf/must-gather-9l8mf"
Jul 08 07:23:14.808644 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:14.808571    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74t4g\" (UniqueName: \"kubernetes.io/projected/84b74e7a-6168-4e5a-ad50-981250b6bea6-kube-api-access-74t4g\") pod \"must-gather-9l8mf\" (UID: \"84b74e7a-6168-4e5a-ad50-981250b6bea6\") " pod="openshift-must-gather-jdzkf/must-gather-9l8mf"
Jul 08 07:23:14.808644 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:14.808620    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/84b74e7a-6168-4e5a-ad50-981250b6bea6-must-gather-output\") pod \"must-gather-9l8mf\" (UID: \"84b74e7a-6168-4e5a-ad50-981250b6bea6\") " pod="openshift-must-gather-jdzkf/must-gather-9l8mf"
Jul 08 07:23:14.808934 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:14.808918    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/84b74e7a-6168-4e5a-ad50-981250b6bea6-must-gather-output\") pod \"must-gather-9l8mf\" (UID: \"84b74e7a-6168-4e5a-ad50-981250b6bea6\") " pod="openshift-must-gather-jdzkf/must-gather-9l8mf"
Jul 08 07:23:14.817195 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:14.817171    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74t4g\" (UniqueName: \"kubernetes.io/projected/84b74e7a-6168-4e5a-ad50-981250b6bea6-kube-api-access-74t4g\") pod \"must-gather-9l8mf\" (UID: \"84b74e7a-6168-4e5a-ad50-981250b6bea6\") " pod="openshift-must-gather-jdzkf/must-gather-9l8mf"
Jul 08 07:23:14.856318 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:14.856292    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jdzkf/must-gather-9l8mf"
Jul 08 07:23:14.980332 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:14.980307    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jdzkf/must-gather-9l8mf"]
Jul 08 07:23:14.982809 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:23:14.982784    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84b74e7a_6168_4e5a_ad50_981250b6bea6.slice/crio-53da4b81d1c79ae7b4b798fbaa95ac3efe398a6857c8e9670bd8a2446f931561 WatchSource:0}: Error finding container 53da4b81d1c79ae7b4b798fbaa95ac3efe398a6857c8e9670bd8a2446f931561: Status 404 returned error can't find the container with id 53da4b81d1c79ae7b4b798fbaa95ac3efe398a6857c8e9670bd8a2446f931561
Jul 08 07:23:15.607632 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:15.607600    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdzkf/must-gather-9l8mf" event={"ID":"84b74e7a-6168-4e5a-ad50-981250b6bea6","Type":"ContainerStarted","Data":"53da4b81d1c79ae7b4b798fbaa95ac3efe398a6857c8e9670bd8a2446f931561"}
Jul 08 07:23:16.615181 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:16.615123    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdzkf/must-gather-9l8mf" event={"ID":"84b74e7a-6168-4e5a-ad50-981250b6bea6","Type":"ContainerStarted","Data":"0d6dffeb959b45e603b82c3cad68aec41c416ad35d8c10652db771b708e47292"}
Jul 08 07:23:16.615666 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:16.615624    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdzkf/must-gather-9l8mf" event={"ID":"84b74e7a-6168-4e5a-ad50-981250b6bea6","Type":"ContainerStarted","Data":"d996763ca9251a6d704fcca6ddce38a14f4513853078eefcb41fcc2abde1c1dd"}
Jul 08 07:23:16.634727 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:16.634680    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jdzkf/must-gather-9l8mf" podStartSLOduration=1.9045982160000001 podStartE2EDuration="2.634665341s" totalImagesPullingTime="730.067125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 07:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:23:16.632908722 +0000 UTC m=+1729.369253436" watchObservedRunningTime="2026-07-08 07:23:16.634665341 +0000 UTC m=+1729.371010054"
Jul 08 07:23:17.108373 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:17.108336    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7zf27_39d02a2c-8802-46ce-a765-576cca29608e/global-pull-secret-syncer/0.log"
Jul 08 07:23:17.188496 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:17.188466    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m5n5f_e3b35714-946f-4024-8c67-ca8e4994f3a2/konnectivity-agent/0.log"
Jul 08 07:23:17.264379 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:17.264347    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-211.ec2.internal_bea9ecd901e1fb22501c8126e0da3e1b/haproxy/0.log"
Jul 08 07:23:19.383918 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:19.383876    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-2mqp7/must-gather-6qb6p"]
Jul 08 07:23:19.384391 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:19.384097    2557 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-2mqp7/must-gather-6qb6p" podUID="652b1071-2ed0-4e41-8095-dcb11d212b99" containerName="copy" containerID="cri-o://63048b640015c74f5e849fef2de08fc417a234ca6a644e1a7395bd6683fc7dfa" gracePeriod=2
Jul 08 07:23:19.388340 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:19.388104    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-2mqp7/must-gather-6qb6p"]
Jul 08 07:23:19.388775 ip-10-0-135-211 kubenswrapper[2557]: E0708 07:23:19.388501    2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6qb6p\" is forbidden: User \"system:node:ip-10-0-135-211.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2mqp7\": no relationship found between node 'ip-10-0-135-211.ec2.internal' and this object" podUID="652b1071-2ed0-4e41-8095-dcb11d212b99" pod="openshift-must-gather-2mqp7/must-gather-6qb6p"
Jul 08 07:23:19.668051 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:19.667030    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2mqp7_must-gather-6qb6p_652b1071-2ed0-4e41-8095-dcb11d212b99/copy/0.log"
Jul 08 07:23:19.668051 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:19.667454    2557 generic.go:356] "Generic (PLEG): container finished" podID="652b1071-2ed0-4e41-8095-dcb11d212b99" containerID="63048b640015c74f5e849fef2de08fc417a234ca6a644e1a7395bd6683fc7dfa" exitCode=143
Jul 08 07:23:19.802238 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:19.802146    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2mqp7_must-gather-6qb6p_652b1071-2ed0-4e41-8095-dcb11d212b99/copy/0.log"
Jul 08 07:23:19.809818 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:19.809690    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2mqp7/must-gather-6qb6p"
Jul 08 07:23:19.959184 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:19.959090    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/652b1071-2ed0-4e41-8095-dcb11d212b99-must-gather-output\") pod \"652b1071-2ed0-4e41-8095-dcb11d212b99\" (UID: \"652b1071-2ed0-4e41-8095-dcb11d212b99\") "
Jul 08 07:23:19.959345 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:19.959202    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxsm7\" (UniqueName: \"kubernetes.io/projected/652b1071-2ed0-4e41-8095-dcb11d212b99-kube-api-access-jxsm7\") pod \"652b1071-2ed0-4e41-8095-dcb11d212b99\" (UID: \"652b1071-2ed0-4e41-8095-dcb11d212b99\") "
Jul 08 07:23:19.960873 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:19.960833    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652b1071-2ed0-4e41-8095-dcb11d212b99-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "652b1071-2ed0-4e41-8095-dcb11d212b99" (UID: "652b1071-2ed0-4e41-8095-dcb11d212b99"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 07:23:19.962285 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:19.962259    2557 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652b1071-2ed0-4e41-8095-dcb11d212b99-kube-api-access-jxsm7" (OuterVolumeSpecName: "kube-api-access-jxsm7") pod "652b1071-2ed0-4e41-8095-dcb11d212b99" (UID: "652b1071-2ed0-4e41-8095-dcb11d212b99"). InnerVolumeSpecName "kube-api-access-jxsm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 07:23:20.059988 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:20.059953    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxsm7\" (UniqueName: \"kubernetes.io/projected/652b1071-2ed0-4e41-8095-dcb11d212b99-kube-api-access-jxsm7\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:23:20.060206 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:20.060190    2557 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/652b1071-2ed0-4e41-8095-dcb11d212b99-must-gather-output\") on node \"ip-10-0-135-211.ec2.internal\" DevicePath \"\""
Jul 08 07:23:20.675528 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:20.675459    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2mqp7_must-gather-6qb6p_652b1071-2ed0-4e41-8095-dcb11d212b99/copy/0.log"
Jul 08 07:23:20.676574 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:20.676554    2557 scope.go:117] "RemoveContainer" containerID="63048b640015c74f5e849fef2de08fc417a234ca6a644e1a7395bd6683fc7dfa"
Jul 08 07:23:20.676923 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:20.676909    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2mqp7/must-gather-6qb6p"
Jul 08 07:23:20.711723 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:20.711696    2557 scope.go:117] "RemoveContainer" containerID="2d6a95d96addd08ca72aee90f016f5483075636a2d3028e8cef20f7da9b451e6"
Jul 08 07:23:21.379425 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.379387    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-dpkd6_a95c5ec7-df5b-4792-b40d-f43d9cf90ba8/cluster-monitoring-operator/0.log"
Jul 08 07:23:21.409465 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.409402    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-jgp8h_03a94a66-03fd-41ab-924f-2ce4f45950f5/kube-state-metrics/0.log"
Jul 08 07:23:21.434282 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.434249    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-jgp8h_03a94a66-03fd-41ab-924f-2ce4f45950f5/kube-rbac-proxy-main/0.log"
Jul 08 07:23:21.458503 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.458478    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-jgp8h_03a94a66-03fd-41ab-924f-2ce4f45950f5/kube-rbac-proxy-self/0.log"
Jul 08 07:23:21.517047 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.517014    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-thfcf_8e6e7393-0c21-4dc5-9975-d7cba956cbae/monitoring-plugin/0.log"
Jul 08 07:23:21.548312 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.548240    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kqz2g_36affcb4-d312-4286-8178-fc47802561ee/node-exporter/0.log"
Jul 08 07:23:21.577425 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.577390    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kqz2g_36affcb4-d312-4286-8178-fc47802561ee/kube-rbac-proxy/0.log"
Jul 08 07:23:21.600045 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.600013    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kqz2g_36affcb4-d312-4286-8178-fc47802561ee/init-textfile/0.log"
Jul 08 07:23:21.788544 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.788505    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="652b1071-2ed0-4e41-8095-dcb11d212b99" path="/var/lib/kubelet/pods/652b1071-2ed0-4e41-8095-dcb11d212b99/volumes"
Jul 08 07:23:21.793480 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.793455    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7dzx6_f258c60f-a3ae-4cce-9105-07d2d3be1ffb/kube-rbac-proxy-main/0.log"
Jul 08 07:23:21.817569 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.817517    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7dzx6_f258c60f-a3ae-4cce-9105-07d2d3be1ffb/kube-rbac-proxy-self/0.log"
Jul 08 07:23:21.841584 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.841544    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-7dzx6_f258c60f-a3ae-4cce-9105-07d2d3be1ffb/openshift-state-metrics/0.log"
Jul 08 07:23:21.884523 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.884492    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7d86829a-db91-4736-aba9-4810965b0ae2/prometheus/0.log"
Jul 08 07:23:21.905735 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.905702    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7d86829a-db91-4736-aba9-4810965b0ae2/config-reloader/0.log"
Jul 08 07:23:21.929512 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.929481    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7d86829a-db91-4736-aba9-4810965b0ae2/thanos-sidecar/0.log"
Jul 08 07:23:21.951642 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.951615    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7d86829a-db91-4736-aba9-4810965b0ae2/kube-rbac-proxy-web/0.log"
Jul 08 07:23:21.988035 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:21.988001    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7d86829a-db91-4736-aba9-4810965b0ae2/kube-rbac-proxy/0.log"
Jul 08 07:23:22.017013 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:22.016954    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7d86829a-db91-4736-aba9-4810965b0ae2/kube-rbac-proxy-thanos/0.log"
Jul 08 07:23:22.044333 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:22.044257    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7d86829a-db91-4736-aba9-4810965b0ae2/init-config-reloader/0.log"
Jul 08 07:23:22.079189 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:22.079153    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-rnvmv_64bf6568-31cf-4b84-855b-967019511798/prometheus-operator/0.log"
Jul 08 07:23:22.098954 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:22.098915    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-rnvmv_64bf6568-31cf-4b84-855b-967019511798/kube-rbac-proxy/0.log"
Jul 08 07:23:22.124413 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:22.124387    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-jvth8_c115bfa0-b765-4f8b-a141-95b37f217b7c/prometheus-operator-admission-webhook/0.log"
Jul 08 07:23:25.497853 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.497823    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"]
Jul 08 07:23:25.498421 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.498294    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652b1071-2ed0-4e41-8095-dcb11d212b99" containerName="copy"
Jul 08 07:23:25.498421 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.498313    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="652b1071-2ed0-4e41-8095-dcb11d212b99" containerName="copy"
Jul 08 07:23:25.498421 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.498326    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652b1071-2ed0-4e41-8095-dcb11d212b99" containerName="gather"
Jul 08 07:23:25.498421 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.498335    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="652b1071-2ed0-4e41-8095-dcb11d212b99" containerName="gather"
Jul 08 07:23:25.498421 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.498420    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="652b1071-2ed0-4e41-8095-dcb11d212b99" containerName="gather"
Jul 08 07:23:25.498681 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.498432    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="652b1071-2ed0-4e41-8095-dcb11d212b99" containerName="copy"
Jul 08 07:23:25.502007 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.501982    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.509000 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.508968    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"]
Jul 08 07:23:25.619902 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.619870    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-podres\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.620078 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.619920    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-lib-modules\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.620078 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.619990    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vmrc\" (UniqueName: \"kubernetes.io/projected/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-kube-api-access-7vmrc\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.620078 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.620039    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-proc\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.620228 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.620111    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-sys\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.721906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.720947    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmrc\" (UniqueName: \"kubernetes.io/projected/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-kube-api-access-7vmrc\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.721906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.721008    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-proc\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.721906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.721065    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-sys\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.721906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.721103    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-podres\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.721906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.721133    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-lib-modules\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.721906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.721325    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-lib-modules\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.721906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.721699    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-proc\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.721906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.721779    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-sys\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.721906 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.721874    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-podres\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.730456 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.730426    2557 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vmrc\" (UniqueName: \"kubernetes.io/projected/a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7-kube-api-access-7vmrc\") pod \"perf-node-gather-daemonset-wqqbr\" (UID: \"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7\") " pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.816286 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.816208    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:25.982525 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:25.982496    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"]
Jul 08 07:23:25.986343 ip-10-0-135-211 kubenswrapper[2557]: W0708 07:23:25.986311    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda2aa13e3_6ba2_4ec9_a3f8_9f035cd0c6d7.slice/crio-928ba4533ac5667adcf974f717b55b5dcbe1957defa60152e9f220d9336a57bc WatchSource:0}: Error finding container 928ba4533ac5667adcf974f717b55b5dcbe1957defa60152e9f220d9336a57bc: Status 404 returned error can't find the container with id 928ba4533ac5667adcf974f717b55b5dcbe1957defa60152e9f220d9336a57bc
Jul 08 07:23:26.184576 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:26.184545    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gtnj4_031afabe-d529-4539-b152-d45a7c651d42/dns/0.log"
Jul 08 07:23:26.204387 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:26.204358    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gtnj4_031afabe-d529-4539-b152-d45a7c651d42/kube-rbac-proxy/0.log"
Jul 08 07:23:26.314819 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:26.314790    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6gzn7_a4f29e73-17e3-42cd-8bb5-980dfe3729c2/dns-node-resolver/0.log"
Jul 08 07:23:26.712818 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:26.712780    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr" event={"ID":"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7","Type":"ContainerStarted","Data":"feb150bb35789cb6462d7a32ee315df4cfc1fd3c3f4a119d3c0a42b57147492c"}
Jul 08 07:23:26.713310 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:26.712844    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr" event={"ID":"a2aa13e3-6ba2-4ec9-a3f8-9f035cd0c6d7","Type":"ContainerStarted","Data":"928ba4533ac5667adcf974f717b55b5dcbe1957defa60152e9f220d9336a57bc"}
Jul 08 07:23:26.713310 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:26.712898    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:26.851518 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:26.851488    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d5qzh_fd567957-630a-4c89-8890-7abcf95e7298/node-ca/0.log"
Jul 08 07:23:28.024140 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:28.024105    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lcn7q_4847909b-7f1a-44f2-9f42-d017286c2d6d/serve-healthcheck-canary/0.log"
Jul 08 07:23:28.392868 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:28.392776    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-5hz6v_f78e9976-f78f-4aca-a985-f2e3d54dbc64/insights-operator/1.log"
Jul 08 07:23:28.393039 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:28.392990    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-5hz6v_f78e9976-f78f-4aca-a985-f2e3d54dbc64/insights-operator/0.log"
Jul 08 07:23:28.534238 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:28.534216    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vbzqz_c4e676c1-8726-4666-b3b7-23eb52cb32a1/kube-rbac-proxy/0.log"
Jul 08 07:23:28.552730 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:28.552704    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vbzqz_c4e676c1-8726-4666-b3b7-23eb52cb32a1/exporter/0.log"
Jul 08 07:23:28.572768 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:28.572739    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vbzqz_c4e676c1-8726-4666-b3b7-23eb52cb32a1/extractor/0.log"
Jul 08 07:23:30.682647 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:30.682621    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-v5bth_cfe911c4-e4c8-46fd-972b-c751108aaf08/s3-init/0.log"
Jul 08 07:23:30.698347 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:30.698326    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-v5bth_cfe911c4-e4c8-46fd-972b-c751108aaf08/download-hf-model/0.log"
Jul 08 07:23:32.728811 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:32.728784    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr"
Jul 08 07:23:32.750985 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:32.750930    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jdzkf/perf-node-gather-daemonset-wqqbr" podStartSLOduration=7.750916234 podStartE2EDuration="7.750916234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 07:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 07:23:26.729902056 +0000 UTC m=+1739.466246773" watchObservedRunningTime="2026-07-08 07:23:32.750916234 +0000 UTC m=+1745.487260948"
Jul 08 07:23:34.475872 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:34.475842    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-7d5tw_fd6db342-303a-4506-bcc3-5916d051cc03/migrator/0.log"
Jul 08 07:23:34.495589 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:34.495558    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-7d5tw_fd6db342-303a-4506-bcc3-5916d051cc03/graceful-termination/0.log"
Jul 08 07:23:34.864609 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:34.864562    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-wpkpv_1617fdda-eaa0-41c3-b37f-220f8f43c63e/kube-storage-version-migrator-operator/1.log"
Jul 08 07:23:34.865605 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:34.865578    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-wpkpv_1617fdda-eaa0-41c3-b37f-220f8f43c63e/kube-storage-version-migrator-operator/0.log"
Jul 08 07:23:35.807426 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:35.807360    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9lcnh_d6e67cb8-a56a-4787-b0cb-a95cab8a760a/kube-multus/0.log"
Jul 08 07:23:35.979310 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:35.979283    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-crwf7_56214b24-626b-4361-965b-b6b26940ac5d/kube-multus-additional-cni-plugins/0.log"
Jul 08 07:23:35.998862 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:35.998829    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-crwf7_56214b24-626b-4361-965b-b6b26940ac5d/egress-router-binary-copy/0.log"
Jul 08 07:23:36.017700 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:36.017671    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-crwf7_56214b24-626b-4361-965b-b6b26940ac5d/cni-plugins/0.log"
Jul 08 07:23:36.037292 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:36.037266    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-crwf7_56214b24-626b-4361-965b-b6b26940ac5d/bond-cni-plugin/0.log"
Jul 08 07:23:36.056211 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:36.056181    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-crwf7_56214b24-626b-4361-965b-b6b26940ac5d/routeoverride-cni/0.log"
Jul 08 07:23:36.075453 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:36.075394    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-crwf7_56214b24-626b-4361-965b-b6b26940ac5d/whereabouts-cni-bincopy/0.log"
Jul 08 07:23:36.095266 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:36.095246    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-crwf7_56214b24-626b-4361-965b-b6b26940ac5d/whereabouts-cni/0.log"
Jul 08 07:23:36.407284 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:36.407199    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r4hnb_77090b45-e309-490a-973e-28c723351e73/network-metrics-daemon/0.log"
Jul 08 07:23:36.430143 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:36.430117    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r4hnb_77090b45-e309-490a-973e-28c723351e73/kube-rbac-proxy/0.log"
Jul 08 07:23:37.377418 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:37.377389    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wkwl_c3f86a51-e751-4c0d-9958-611e61cdf7b0/ovn-controller/0.log"
Jul 08 07:23:37.395541 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:37.395478    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wkwl_c3f86a51-e751-4c0d-9958-611e61cdf7b0/ovn-acl-logging/0.log"
Jul 08 07:23:37.421372 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:37.421330    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wkwl_c3f86a51-e751-4c0d-9958-611e61cdf7b0/kube-rbac-proxy-node/0.log"
Jul 08 07:23:37.440567 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:37.440547    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wkwl_c3f86a51-e751-4c0d-9958-611e61cdf7b0/kube-rbac-proxy-ovn-metrics/0.log"
Jul 08 07:23:37.457621 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:37.457600    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wkwl_c3f86a51-e751-4c0d-9958-611e61cdf7b0/northd/0.log"
Jul 08 07:23:37.476956 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:37.476939    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wkwl_c3f86a51-e751-4c0d-9958-611e61cdf7b0/nbdb/0.log"
Jul 08 07:23:37.499854 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:37.499838    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wkwl_c3f86a51-e751-4c0d-9958-611e61cdf7b0/sbdb/0.log"
Jul 08 07:23:37.541121 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:37.541102    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8wkwl_c3f86a51-e751-4c0d-9958-611e61cdf7b0/ovnkube-controller/0.log"
Jul 08 07:23:38.654722 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:38.654691    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-989f8b8dc-mwb6f_372d33b3-653f-42b0-a478-538a4b2f7340/check-endpoints/0.log"
Jul 08 07:23:38.704287 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:38.704262    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-h4bmr_8de691b9-d68b-4ae5-a0aa-47587f146d7e/network-check-target-container/0.log"
Jul 08 07:23:39.568223 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:39.568193    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-xj4hz_5b951e1f-2e78-4d1f-af51-eedaabc4a02a/iptables-alerter/0.log"
Jul 08 07:23:40.188461 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:40.188439    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-stt62_0d573e55-4edc-47fb-abeb-7c58ddb30139/tuned/0.log"
Jul 08 07:23:41.885105 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:41.885038    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-56fcf86bf-7lsvs_74b284c7-3b36-48fc-9c30-077a20898d05/cluster-samples-operator/0.log"
Jul 08 07:23:41.900051 ip-10-0-135-211 kubenswrapper[2557]: I0708 07:23:41.900025    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-56fcf86bf-7lsvs_74b284c7-3b36-48fc-9c30-077a20898d05/cluster-samples-operator-watch/0.log"