Sep 09 18:31:49.993531 ip-10-0-134-63 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 18:31:50.434535 ip-10-0-134-63 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 18:31:50.434535 ip-10-0-134-63 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 18:31:50.434535 ip-10-0-134-63 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 18:31:50.434535 ip-10-0-134-63 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 18:31:50.434535 ip-10-0-134-63 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 18:31:50.435559 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.435464    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 18:31:50.440893 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440863    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 18:31:50.440893 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440889    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 18:31:50.440893 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440895    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 18:31:50.440893 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440898    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440902    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440906    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440909    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440912    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440915    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440918    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440921    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440925    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440930    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440933    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440937    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440940    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440943    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440946    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440949    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440953    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440963    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440966    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440969    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 18:31:50.441028 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440972    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440975    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440978    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440980    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440983    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440986    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440988    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440992    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440995    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.440997    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441000    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441003    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441006    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441009    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441011    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441014    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441017    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441021    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441025    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441028    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 18:31:50.441526 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441031    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441035    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441038    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441041    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441045    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441047    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441050    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441053    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441055    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441059    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441061    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441064    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441067    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441070    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441072    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441075    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441078    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441081    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441083    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441086    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 18:31:50.442016 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441089    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441092    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441095    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441097    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441100    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441104    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441106    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441109    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441111    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441114    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441116    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441120    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441123    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441125    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441129    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441131    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441134    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441137    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441139    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441141    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 18:31:50.442514 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441144    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441161    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441163    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441166    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441169    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441171    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441174    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441178    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441182    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441185    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441188    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441192    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441195    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441198    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441201    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441204    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441975    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441983    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441987    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 18:31:50.443072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441991    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441994    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.441997    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442000    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442003    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442005    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442008    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442011    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442015    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442017    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442021    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442025    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442029    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442032    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442035    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442038    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442040    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442043    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442046    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 18:31:50.443594 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442048    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442051    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442054    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442056    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442059    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442063    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442066    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442069    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442072    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442074    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442077    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442080    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442082    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442085    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442088    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442091    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442095    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442098    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442101    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442104    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 18:31:50.444090 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442107    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442110    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442113    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442116    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442119    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442122    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442124    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442126    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442129    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442132    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442134    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442137    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442140    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442143    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442160    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442163    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442166    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442169    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442171    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442174    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 18:31:50.444640 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442177    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442180    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442183    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442186    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442188    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442191    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442194    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442197    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442199    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442202    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442204    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442207    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442210    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442212    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442215    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442218    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442221    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442224    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442226    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442229    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 18:31:50.445162 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442231    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442234    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442236    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442239    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442241    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442244    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442247    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442250    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442252    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442255    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442257    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442260    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442263    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442265    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442268    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442271    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442274    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442680    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442685    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 18:31:50.445665 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442688    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442691    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442694    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442697    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442700    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442703    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442705    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442708    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442711    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442714    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442717    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442720    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442722    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442725    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442728    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442733    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442736    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442739    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442745    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442748    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 18:31:50.446133 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442751    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442754    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442757    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442760    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442763    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442765    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442768    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442772    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442775    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442777    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442780    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442783    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442786    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442788    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442791    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442793    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442796    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442799    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442801    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442804    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 18:31:50.446651 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442806    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442810    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442813    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442815    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442818    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442821    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442823    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442825    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442828    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442831    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442835    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442837    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442840    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442843    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442845    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442848    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442853    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442856    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442861    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442864    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 18:31:50.447174 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442867    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442870    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442873    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442875    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442878    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442881    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442883    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442886    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442888    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442891    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442893    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442896    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442898    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442901    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442904    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442909    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442911    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442914    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442916    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442920    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 18:31:50.447681 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442923    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442925    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442930    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442933    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442935    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442938    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442940    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442943    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442946    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442949    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442953    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442955    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442958    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442961    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442963    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442966    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.442968    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443058    2575 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443110    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443120    2575 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443124    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 18:31:50.448184 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443129    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443132    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443141    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443164    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443168    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443171    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443175    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443179    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443182    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443186    2575 flags.go:64] FLAG: --cgroup-root=""
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443189    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443192    2575 flags.go:64] FLAG: --client-ca-file=""
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443195    2575 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443199    2575 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443204    2575 flags.go:64] FLAG: --cluster-domain=""
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443207    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443210    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443213    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443217    2575 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443221    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443225    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443230    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443233    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443237    2575 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 18:31:50.448753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443240    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443243    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443246    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443249    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443253    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443257    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443260    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443263    2575 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443266    2575 flags.go:64] FLAG: --enable-server="true"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443269    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443275    2575 flags.go:64] FLAG: --event-burst="100"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443278    2575 flags.go:64] FLAG: --event-qps="50"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443281    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443284    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443287    2575 flags.go:64] FLAG: --eviction-hard=""
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443292    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443295    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443298    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443301    2575 flags.go:64] FLAG: --eviction-soft=""
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443304    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443307    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443312    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443315    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443318    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443321    2575 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 18:31:50.449367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443324    2575 flags.go:64] FLAG: --feature-gates=""
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443328    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443331    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443334    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443339    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443342    2575 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443345    2575 flags.go:64] FLAG: --help="false"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443349    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443352    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443355    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443358    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443361    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443365    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443368    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443371    2575 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443374    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443377    2575 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443380    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443383    2575 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443386    2575 flags.go:64] FLAG: --kube-reserved=""
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443389    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443392    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443395    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443405    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 18:31:50.450001 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443409    2575 flags.go:64] FLAG: --lock-file=""
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443412    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443415    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443418    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443430    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443433    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443436    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443439    2575 flags.go:64] FLAG: --logging-format="text"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443442    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443446    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443449    2575 flags.go:64] FLAG: --manifest-url=""
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443452    2575 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443458    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443461    2575 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443465    2575 flags.go:64] FLAG: --max-pods="110"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443468    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443471    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443474    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443477    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443480    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443483    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443486    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443495    2575 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443498    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443502    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 18:31:50.450834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443505    2575 flags.go:64] FLAG: --pod-cidr=""
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443508    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443514    2575 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443517    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443520    2575 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443523    2575 flags.go:64] FLAG: --port="10250"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443526    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443530    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f011299fc821b0bd"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443534    2575 flags.go:64] FLAG: --qos-reserved=""
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443537    2575 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443540    2575 flags.go:64] FLAG: --register-node="true"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443543    2575 flags.go:64] FLAG: --register-with-taints=""
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443550    2575 flags.go:64] FLAG: --registry-burst="10"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443553    2575 flags.go:64] FLAG: --registry-qps="5"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443556    2575 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443559    2575 flags.go:64] FLAG: --reserved-memory=""
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443563    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443567    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443570    2575 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443574    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443577    2575 flags.go:64] FLAG: --runonce="false"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443580    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443584    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443587    2575 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443590    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 18:31:50.451952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443593    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443597    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443601    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443604    2575 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443607    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443610    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443613    2575 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443616    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443619    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443622    2575 flags.go:64] FLAG: --system-cgroups=""
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443625    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443631    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443634    2575 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443636    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443640    2575 flags.go:64] FLAG: --tls-min-version=""
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443644    2575 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443647    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443650    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443654    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443658    2575 flags.go:64] FLAG: --v="2"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443663    2575 flags.go:64] FLAG: --version="false"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443667    2575 flags.go:64] FLAG: --vmodule=""
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443671    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.443674    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443800    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 18:31:50.453088 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443805    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443810    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443813    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443816    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443819    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443821    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443824    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443827    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443830    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443832    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443835    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443838    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443840    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443843    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443846    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443849    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443851    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443854    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443857    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443859    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 18:31:50.454220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443862    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443864    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443867    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443870    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443873    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443876    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443880    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443882    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443885    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443888    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443890    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443893    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443896    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443900    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443902    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443906    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443910    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443913    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443916    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443919    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 18:31:50.454979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443921    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443924    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443926    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443929    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443932    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443934    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443937    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443941    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443945    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443948    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443952    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443955    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443957    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443960    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443962    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443965    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443968    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443972    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443977    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 18:31:50.455769 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443980    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443983    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443986    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443988    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443991    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443994    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.443997    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444000    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444003    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444005    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444008    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444011    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444013    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444016    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444019    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444022    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444025    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444027    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444030    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444033    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 18:31:50.456260 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444035    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444038    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444041    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444044    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444046    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444049    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444052    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444054    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444056    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444059    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444064    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444068    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444071    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444073    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444076    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444079    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444081    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444084    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 18:31:50.456939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.444087    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 18:31:50.457720 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.444097    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 18:31:50.457720 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.452994    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 18:31:50.457720 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.453025    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 18:31:50.457720 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.453056    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 18:31:50.457720 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.453070    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 18:31:50.457720 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453200    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 18:31:50.457720 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453210    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 18:31:50.457720 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453215    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 18:31:50.457720 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453220    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 18:31:50.457720 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453225    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 18:31:50.457720 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453229    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 18:31:50.457720 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453233    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 18:31:50.457720 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453241    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 18:31:50.457720 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453247    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453251    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453255    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453260    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453265    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453270    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453275    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453279    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453283    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453288    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453303    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453307    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453312    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453316    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453321    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453326    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453330    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453335    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453339    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453344    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 18:31:50.458274 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453350    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453355    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453359    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453363    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453367    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453372    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453377    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453381    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453385    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453389    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453394    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453397    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453410    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453414    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453418    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453422    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453426    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453430    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453434    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453438    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 18:31:50.458863 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453442    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453447    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453450    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453462    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453467    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453472    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453480    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453484    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453489    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453493    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453497    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453501    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453506    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453510    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453515    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453519    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453526    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453531    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453535    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 18:31:50.459479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453540    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453545    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453550    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453554    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453559    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453563    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453567    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453571    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453575    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453579    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453583    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453588    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453592    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453596    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453602    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453607    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453612    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453625    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453631    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 18:31:50.460026 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453635    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453639    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453643    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453648    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453652    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453656    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453661    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453666    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453670    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453674    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453679    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453683    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453687    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.453695    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453958    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 18:31:50.460534 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453969    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453977    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453983    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453988    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453993    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.453998    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454003    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454007    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454012    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454017    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454021    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454025    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454029    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454033    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454037    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454041    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454060    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454065    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454069    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454074    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 18:31:50.460980 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454078    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454082    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454086    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454091    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454096    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454100    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454104    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454108    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454112    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454116    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454122    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454128    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454133    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454137    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454142    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454161    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454166    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454170    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454174    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 18:31:50.461512 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454178    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454182    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454186    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454191    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454195    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454199    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454204    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454208    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454212    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454216    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454229    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454233    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454236    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454240    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454244    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454249    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454253    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454257    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454262    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454265    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 18:31:50.461986 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454270    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454274    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454278    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454283    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454287    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454291    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454295    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454300    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454304    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454308    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454312    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454316    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454320    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454324    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454328    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454333    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454337    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454341    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454345    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 18:31:50.462728 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454349    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454354    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454358    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454362    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454375    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454379    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454383    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454388    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454393    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454397    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454401    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454405    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454414    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454419    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454424    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454429    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454433    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454438    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454443    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 18:31:50.463595 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:50.454447    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 18:31:50.464327 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.454455    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 18:31:50.464327 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.454655    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 18:31:50.464327 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.458705    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 18:31:50.464327 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.459811    2575 server.go:1019] "Starting client certificate rotation"
Sep 09 18:31:50.464327 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.459920    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 18:31:50.464327 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.459962    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 18:31:50.464327 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.463172    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 18:31:50.467679 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.467646    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 18:31:50.478308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.478284    2575 log.go:25] "Validated CRI v1 runtime API"
Sep 09 18:31:50.482225 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.482204    2575 log.go:25] "Validated CRI v1 image API"
Sep 09 18:31:50.483881 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.483864    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 18:31:50.486441 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.486415    2575 fs.go:135] Filesystem UUIDs: map[04fbaeb8-c0d4-4ea4-b282-e6dd8594a745:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 dcb21d77-55da-46f9-9e08-4b0d1d7eed5a:/dev/nvme0n1p4]
Sep 09 18:31:50.486547 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.486438    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 18:31:50.490290 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.490267    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 18:31:50.492546 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.492402    2575 manager.go:216] Machine: {Timestamp:2026-09-09 18:31:50.491101912 +0000 UTC m=+0.375821197 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3088353 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2114d207757c380260c0918c3dcfc3 SystemUUID:ec2114d2-0775-7c38-0260-c0918c3dcfc3 BootID:f9b00095-2de3-4e99-be0e-c5c73b1bdd44 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ad:d6:90:49:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ad:d6:90:49:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:63:a8:d0:21:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 18:31:50.492546 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.492543    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 18:31:50.492670 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.492654    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 18:31:50.496490 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.496450    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 18:31:50.496681 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.496492    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-63.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 18:31:50.497402 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.497388    2575 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 18:31:50.497402 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.497403    2575 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 18:31:50.497471 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.497412    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 18:31:50.498358 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.498343    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 18:31:50.499829 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.499814    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 18:31:50.504115 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.504097    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 18:31:50.504277 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.504267    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 18:31:50.504310 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.504301    2575 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 18:31:50.504358 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.504314    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 18:31:50.504358 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.504345    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 18:31:50.504358 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.504357    2575 kubelet.go:398] "Adding apiserver pod source"
Sep 09 18:31:50.504443 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.504367    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 18:31:50.504988 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.504960    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 18:31:50.504988 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.504985    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 18:31:50.508252 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.508233    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 18:31:50.510804 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.510787    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 18:31:50.510867 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.510823    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 18:31:50.510912 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.510894    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6457d"
Sep 09 18:31:50.510988 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.510978    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 18:31:50.511017 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.510994    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 18:31:50.511017 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.511000    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 18:31:50.511017 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.511006    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 18:31:50.511171 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.511027    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 18:31:50.511171 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.511034    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 18:31:50.511639 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.511626    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 18:31:50.511639 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.511638    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 18:31:50.511703 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.511645    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 18:31:50.511703 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.511652    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 18:31:50.511703 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.511668    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 18:31:50.511703 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.511677    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 18:31:50.512423 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.512411    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 18:31:50.512459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.512424    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 18:31:50.516239 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.516206    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6457d"
Sep 09 18:31:50.518056 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.518035    2575 server.go:1268] "Started kubelet"
Sep 09 18:31:50.518187 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.518120    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 18:31:50.518324 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.518302    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-63.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 18:31:50.518375 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.518308    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 18:31:50.518416 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.518384    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-63.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 18:31:50.518486 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.518443    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 18:31:50.518530 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.518510    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 18:31:50.519611 ip-10-0-134-63 systemd[1]: Started Kubernetes Kubelet.
Sep 09 18:31:50.519870 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.519844    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 18:31:50.520916 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.520904    2575 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 18:31:50.524093 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.524071    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 18:31:50.524093 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.524078    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 18:31:50.524256 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.524237    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-63.ec2.internal\" not found"
Sep 09 18:31:50.524769 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.524751    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 18:31:50.524874 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.524768    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 18:31:50.524874 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.524876    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 18:31:50.525041 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.524983    2575 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 18:31:50.525041 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.524991    2575 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 18:31:50.525041 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.525020    2575 factory.go:55] Registering systemd factory
Sep 09 18:31:50.525041 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.525035    2575 factory.go:223] Registration of the systemd container factory successfully
Sep 09 18:31:50.525301 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.525281    2575 factory.go:153] Registering CRI-O factory
Sep 09 18:31:50.525393 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.525316    2575 factory.go:223] Registration of the crio container factory successfully
Sep 09 18:31:50.525393 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.525328    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 18:31:50.525393 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.525381    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 18:31:50.525524 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.525401    2575 factory.go:103] Registering Raw factory
Sep 09 18:31:50.525524 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.525417    2575 manager.go:1196] Started watching for new ooms in manager
Sep 09 18:31:50.525834 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.525823    2575 manager.go:349] Starting recovery of all containers
Sep 09 18:31:50.527553 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.527380    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 18:31:50.530992 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.530955    2575 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-63.ec2.internal\" not found" node="ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.536904 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.536883    2575 manager.go:354] Recovery completed
Sep 09 18:31:50.545007 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.544982    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 18:31:50.547359 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.547334    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 18:31:50.547509 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.547374    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 18:31:50.547509 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.547386    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasSufficientPID"
Sep 09 18:31:50.547997 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.547979    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 18:31:50.547997 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.547994    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 18:31:50.548142 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.548006    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 18:31:50.548142 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.548012    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 18:31:50.548864 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.548846    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 18:31:50.548949 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.548877    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 18:31:50.548949 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.548887    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasSufficientPID"
Sep 09 18:31:50.549480 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.549463    2575 policy_none.go:49] "None policy: Start"
Sep 09 18:31:50.549551 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.549482    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 18:31:50.549551 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.549494    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 18:31:50.550859 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.550845    2575 policy_none.go:47] "Start"
Sep 09 18:31:50.590912 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.590892    2575 manager.go:337] "Starting Device Plugin manager"
Sep 09 18:31:50.605176 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.590948    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 18:31:50.605176 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.590961    2575 server.go:87] "Starting device plugin registration server"
Sep 09 18:31:50.605176 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.591440    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 18:31:50.605176 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.591454    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 18:31:50.605176 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.591557    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 18:31:50.605176 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.591649    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 18:31:50.605176 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.591661    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 18:31:50.605176 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.592285    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 18:31:50.605176 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.592363    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-63.ec2.internal\" not found"
Sep 09 18:31:50.665379 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.665339    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 18:31:50.666627 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.666606    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 18:31:50.666721 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.666671    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 18:31:50.666721 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.666701    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 18:31:50.666830 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.666754    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 18:31:50.669348 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.669314    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 18:31:50.692441 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.692317    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 18:31:50.693354 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.693331    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 18:31:50.693466 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.693370    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 18:31:50.693466 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.693384    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasSufficientPID"
Sep 09 18:31:50.693466 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.693407    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.702705 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.702682    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.702834 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.702711    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-63.ec2.internal\": node \"ip-10-0-134-63.ec2.internal\" not found"
Sep 09 18:31:50.718687 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.718654    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-63.ec2.internal\" not found"
Sep 09 18:31:50.767031 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.766998    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-63.ec2.internal"]
Sep 09 18:31:50.767133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.767094    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 18:31:50.768880 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.768861    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 18:31:50.768998 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.768896    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 18:31:50.768998 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.768909    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasSufficientPID"
Sep 09 18:31:50.771057 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.771042    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 18:31:50.771225 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.771205    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.771331 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.771243    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 18:31:50.771975 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.771958    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 18:31:50.772121 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.771988    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 18:31:50.772121 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.771958    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 18:31:50.772121 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.772003    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasSufficientPID"
Sep 09 18:31:50.772121 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.772025    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 18:31:50.772121 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.772042    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasSufficientPID"
Sep 09 18:31:50.774065 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.774048    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.774181 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.774076    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 18:31:50.776015 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.775999    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 18:31:50.776110 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.776026    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 18:31:50.776110 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.776041    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeHasSufficientPID"
Sep 09 18:31:50.799082 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.799051    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-63.ec2.internal\" not found" node="ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.803769 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.803746    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-63.ec2.internal\" not found" node="ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.819461 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.819424    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-63.ec2.internal\" not found"
Sep 09 18:31:50.826417 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.826381    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b14058b333631407a76dac7ca4cc84a1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-63.ec2.internal\" (UID: \"b14058b333631407a76dac7ca4cc84a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.826539 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.826418    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/763960a7b123d75ea1d5a1fad095cc1b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal\" (UID: \"763960a7b123d75ea1d5a1fad095cc1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.826539 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.826437    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/763960a7b123d75ea1d5a1fad095cc1b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal\" (UID: \"763960a7b123d75ea1d5a1fad095cc1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.919803 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:50.919757    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-63.ec2.internal\" not found"
Sep 09 18:31:50.927161 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.927128    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/763960a7b123d75ea1d5a1fad095cc1b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal\" (UID: \"763960a7b123d75ea1d5a1fad095cc1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.927234 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.927177    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/763960a7b123d75ea1d5a1fad095cc1b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal\" (UID: \"763960a7b123d75ea1d5a1fad095cc1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.927234 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.927193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b14058b333631407a76dac7ca4cc84a1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-63.ec2.internal\" (UID: \"b14058b333631407a76dac7ca4cc84a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.927234 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.927219    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b14058b333631407a76dac7ca4cc84a1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-63.ec2.internal\" (UID: \"b14058b333631407a76dac7ca4cc84a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.927341 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.927221    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/763960a7b123d75ea1d5a1fad095cc1b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal\" (UID: \"763960a7b123d75ea1d5a1fad095cc1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:50.927341 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:50.927216    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/763960a7b123d75ea1d5a1fad095cc1b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal\" (UID: \"763960a7b123d75ea1d5a1fad095cc1b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:51.020691 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:51.020593    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-63.ec2.internal\" not found"
Sep 09 18:31:51.103226 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:51.103190    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:51.106794 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:51.106771    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:51.120981 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:51.120952    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-63.ec2.internal\" not found"
Sep 09 18:31:51.221961 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:51.221915    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-63.ec2.internal\" not found"
Sep 09 18:31:51.322544 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:51.322465    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-63.ec2.internal\" not found"
Sep 09 18:31:51.423141 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:51.423095    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-63.ec2.internal\" not found"
Sep 09 18:31:51.460638 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:51.460599    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 18:31:51.461119 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:51.460778    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 18:31:51.461119 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:51.460806    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 18:31:51.520870 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:51.520828    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 18:26:52 +0000 UTC" deadline="2028-02-28 13:39:22.770433199 +0000 UTC"
Sep 09 18:31:51.520870 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:51.520859    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12883h7m31.249578579s"
Sep 09 18:31:51.523228 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:51.523207    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-63.ec2.internal\" not found"
Sep 09 18:31:51.612484 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:51.612398    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 18:31:51.624247 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:51.624217    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:51.633782 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:51.633757    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 18:31:51.635619 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:51.635600    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-63.ec2.internal"
Sep 09 18:31:51.643306 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:51.643285    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 18:31:51.715134 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:51.715099    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 18:31:51.814423 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:51.814379    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14058b333631407a76dac7ca4cc84a1.slice/crio-b1eb26ec088658ad5e842891b5fe5b95ea8f81440be50a5faf2a3087c09d794f WatchSource:0}: Error finding container b1eb26ec088658ad5e842891b5fe5b95ea8f81440be50a5faf2a3087c09d794f: Status 404 returned error can't find the container with id b1eb26ec088658ad5e842891b5fe5b95ea8f81440be50a5faf2a3087c09d794f
Sep 09 18:31:51.820109 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:51.820085    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 18:31:52.422753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.422718    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 18:31:52.505712 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.505685    2575 apiserver.go:52] "Watching apiserver"
Sep 09 18:31:52.513950 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.513916    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 18:31:52.514514 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.514487    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-fs6d5","openshift-network-operator/iptables-alerter-kdnwx","openshift-ovn-kubernetes/ovnkube-node-srx2c","kube-system/global-pull-secret-syncer-7vhcs","kube-system/kube-apiserver-proxy-ip-10-0-134-63.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jtwwp","openshift-image-registry/node-ca-t75rf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal","openshift-multus/multus-qdrf9","kube-system/konnectivity-agent-jnb9f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw","openshift-dns/node-resolver-lpvdl","openshift-multus/multus-additional-cni-plugins-pmjtg","openshift-multus/network-metrics-daemon-kdnq9"]
Sep 09 18:31:52.517187 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.517160    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t75rf"
Sep 09 18:31:52.519765 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.519675    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.519907 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.519884    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 18:31:52.520355 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.520191    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xs2rh\""
Sep 09 18:31:52.520355 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.520240    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 18:31:52.520355 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.520346    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 18:31:52.521595 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.521558    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 18:31:52.522018 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.521975    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 18:31:52.522483 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.522459    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tjf6c\""
Sep 09 18:31:52.522723 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.522704    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 18:31:52.522802 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.522759    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 18:31:52.522984 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.522965    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.525691 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.525669    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 18:31:52.525870 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.525848    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 18:31:52.525870 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.525860    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-879b5\""
Sep 09 18:31:52.526058 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.526043    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 18:31:52.526106 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.526096    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 18:31:52.526227 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.525670    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 18:31:52.526287 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.526250    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 18:31:52.526434 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.526418    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:31:52.526506 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.526450    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 18:31:52.526506 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:52.526488    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:31:52.527990 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.527879    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:31:52.527990 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:52.527950    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:31:52.530551 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.530529    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.533270 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.533044    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.533386 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.533289    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 18:31:52.533386 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.533297    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-plr25\""
Sep 09 18:31:52.533556 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.533538    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 18:31:52.534747 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.534719    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-etc-openvswitch\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.534862 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.534764    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-cni-bin\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.534862 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.534797    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b505067-ec95-4acc-bc12-554d0512edb2-env-overrides\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.535086 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.534870    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b505067-ec95-4acc-bc12-554d0512edb2-ovnkube-script-lib\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.535086 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.534908    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:31:52.535086 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.534986    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-kubelet\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.535086 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535033    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-slash\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.535086 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535066    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-run-netns\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.535369 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535127    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf7kr\" (UniqueName: \"kubernetes.io/projected/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-kube-api-access-qf7kr\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.535369 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535178    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-run-ovn\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.535369 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535204    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-cni-netd\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.535369 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535231    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.535369 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535268    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-socket-dir\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.535369 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535300    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-registration-dir\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.535369 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535366    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-run-systemd\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.535684 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535397    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7csw\" (UniqueName: \"kubernetes.io/projected/6b505067-ec95-4acc-bc12-554d0512edb2-kube-api-access-g7csw\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.535684 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535423    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3526bb20-1563-4c64-970b-e159d015bd3d-kubelet-config\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:31:52.535684 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535454    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.535684 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-var-lib-openvswitch\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.535883 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535862    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-log-socket\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.535935 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535901    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b505067-ec95-4acc-bc12-554d0512edb2-ovn-node-metrics-cert\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.535980 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.535932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1753a297-ac2b-49cd-b9c9-459465d3d912-host\") pod \"node-ca-t75rf\" (UID: \"1753a297-ac2b-49cd-b9c9-459465d3d912\") " pod="openshift-image-registry/node-ca-t75rf"
Sep 09 18:31:52.536178 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536141    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-systemd-units\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.536241 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536203    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-node-log\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.536360 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536330    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b505067-ec95-4acc-bc12-554d0512edb2-ovnkube-config\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.536970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536474    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g5zzl\""
Sep 09 18:31:52.536970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536503    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 18:31:52.536970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536564    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 18:31:52.536970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536568    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-run-ovn-kubernetes\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.536970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536726    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3526bb20-1563-4c64-970b-e159d015bd3d-dbus\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:31:52.536970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536752    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-device-dir\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.536970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536774    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 18:31:52.536970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536799    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 18:31:52.536970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536799    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbqc9\" (UniqueName: \"kubernetes.io/projected/1753a297-ac2b-49cd-b9c9-459465d3d912-kube-api-access-rbqc9\") pod \"node-ca-t75rf\" (UID: \"1753a297-ac2b-49cd-b9c9-459465d3d912\") " pod="openshift-image-registry/node-ca-t75rf"
Sep 09 18:31:52.536970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536849    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.536970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536888    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-run-openvswitch\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.536970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536933    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1753a297-ac2b-49cd-b9c9-459465d3d912-serviceca\") pod \"node-ca-t75rf\" (UID: \"1753a297-ac2b-49cd-b9c9-459465d3d912\") " pod="openshift-image-registry/node-ca-t75rf"
Sep 09 18:31:52.536970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.536964    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-sys-fs\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.537672 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.537205    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:52.539719 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.539503    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vgbjg\""
Sep 09 18:31:52.539719 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.539511    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 18:31:52.540262 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.539931    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kdnwx"
Sep 09 18:31:52.540262 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.539964    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 18:31:52.542712 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.542397    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 18:31:52.542712 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.542446    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 18:31:52.542712 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.542510    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 18:31:52.542712 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.542608    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2jx58\""
Sep 09 18:31:52.542957 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.542795    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 18:31:52.542957 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.542874    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jnb9f"
Sep 09 18:31:52.545286 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.545070    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 18:31:52.545286 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.545135    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 18:31:52.545286 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.545172    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:31:52.545286 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:52.545245    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:31:52.545689 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.545669    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xjcql\""
Sep 09 18:31:52.547659 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.547637    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lpvdl"
Sep 09 18:31:52.549941 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.549918    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 18:31:52.550190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.550171    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 18:31:52.550296 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.550273    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9l728\""
Sep 09 18:31:52.560337 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.560304    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9s5wf"
Sep 09 18:31:52.567804 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.567768    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9s5wf"
Sep 09 18:31:52.626262 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.626232    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 18:31:52.638230 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-cni-netd\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.638461 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638243    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c717df31-e04f-4681-8af3-3c322acadca5-host-slash\") pod \"iptables-alerter-kdnwx\" (UID: \"c717df31-e04f-4681-8af3-3c322acadca5\") " pod="openshift-network-operator/iptables-alerter-kdnwx"
Sep 09 18:31:52.638461 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638282    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-cnibin\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.638461 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638300    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-run-k8s-cni-cncf-io\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.638461 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638327    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-cni-netd\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.638461 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638324    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-var-lib-kubelet\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.638461 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638408    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-socket-dir\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.638461 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-run-systemd\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.638763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638494    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-run-systemd\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.638763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3526bb20-1563-4c64-970b-e159d015bd3d-kubelet-config\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:31:52.638763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638547    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5ae7904-5d7e-48dc-bb78-af622de153e4-cni-binary-copy\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.638763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-tuned\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.638763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638602    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-var-lib-openvswitch\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.638763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638624    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-log-socket\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.638763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638643    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tbfw\" (UniqueName: \"kubernetes.io/projected/d5ae7904-5d7e-48dc-bb78-af622de153e4-kube-api-access-5tbfw\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.638763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638668    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-systemd-units\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.638763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638678    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3526bb20-1563-4c64-970b-e159d015bd3d-kubelet-config\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:31:52.638763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638692    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b505067-ec95-4acc-bc12-554d0512edb2-ovnkube-config\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.638763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638726    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-modprobe-d\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.638763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638746    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-log-socket\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.638763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638750    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-sys\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638776    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-var-lib-kubelet\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638802    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3526bb20-1563-4c64-970b-e159d015bd3d-dbus\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638814    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-var-lib-openvswitch\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-var-lib-cni-bin\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638879    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-cni-binary-copy\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638904    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcd4d\" (UniqueName: \"kubernetes.io/projected/d4099f20-afa3-482b-a22e-c4d59554f7fd-kube-api-access-qcd4d\") pod \"node-resolver-lpvdl\" (UID: \"d4099f20-afa3-482b-a22e-c4d59554f7fd\") " pod="openshift-dns/node-resolver-lpvdl"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638940    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3526bb20-1563-4c64-970b-e159d015bd3d-dbus\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638949    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.638993    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-run-openvswitch\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639025    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-cnibin\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639050    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-host\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639086    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-etc-kubernetes\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639133    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1753a297-ac2b-49cd-b9c9-459465d3d912-serviceca\") pod \"node-ca-t75rf\" (UID: \"1753a297-ac2b-49cd-b9c9-459465d3d912\") " pod="openshift-image-registry/node-ca-t75rf"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639173    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-etc-openvswitch\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639201    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639205    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-run-openvswitch\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.639308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639228    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-system-cni-dir\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639170    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-socket-dir\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639342    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b505067-ec95-4acc-bc12-554d0512edb2-ovnkube-config\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639350    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639361    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-systemd-units\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639389    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-etc-openvswitch\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639416    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-multus-conf-dir\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:52.639470    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639537    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbhg5\" (UniqueName: \"kubernetes.io/projected/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-kube-api-access-jbhg5\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:52.639563    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret podName:3526bb20-1563-4c64-970b-e159d015bd3d nodeName:}" failed. No retries permitted until 2026-09-09 18:31:53.139520885 +0000 UTC m=+3.024240165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret") pod "global-pull-secret-syncer-7vhcs" (UID: "3526bb20-1563-4c64-970b-e159d015bd3d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639623    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzpht\" (UniqueName: \"kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht\") pod \"network-check-target-fs6d5\" (UID: \"e27817cd-3adb-43ef-8434-7d43af58c903\") " pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639649    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-slash\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639671    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-run-netns\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639701    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-hostroot\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639724    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639726    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-run-netns\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639744    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-sysctl-conf\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.640026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639760    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-slash\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639765    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-lib-modules\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639803    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf7kr\" (UniqueName: \"kubernetes.io/projected/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-kube-api-access-qf7kr\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639843    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-run-ovn\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.639976    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640020    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c717df31-e04f-4681-8af3-3c322acadca5-iptables-alerter-script\") pod \"iptables-alerter-kdnwx\" (UID: \"c717df31-e04f-4681-8af3-3c322acadca5\") " pod="openshift-network-operator/iptables-alerter-kdnwx"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640062    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-run-netns\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640063    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640092    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640116    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1753a297-ac2b-49cd-b9c9-459465d3d912-serviceca\") pod \"node-ca-t75rf\" (UID: \"1753a297-ac2b-49cd-b9c9-459465d3d912\") " pod="openshift-image-registry/node-ca-t75rf"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640120    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-run-ovn\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640163    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-run\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-registration-dir\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640253    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7csw\" (UniqueName: \"kubernetes.io/projected/6b505067-ec95-4acc-bc12-554d0512edb2-kube-api-access-g7csw\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640292    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c58852df-cacb-4054-8f99-18344f10936d-agent-certs\") pod \"konnectivity-agent-jnb9f\" (UID: \"c58852df-cacb-4054-8f99-18344f10936d\") " pod="kube-system/konnectivity-agent-jnb9f"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640305    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-registration-dir\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.640838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640397    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640440    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b505067-ec95-4acc-bc12-554d0512edb2-ovn-node-metrics-cert\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640496    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66bae284-0ba9-48e6-af3e-7deafe38260e-tmp\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640506    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640524    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1753a297-ac2b-49cd-b9c9-459465d3d912-host\") pod \"node-ca-t75rf\" (UID: \"1753a297-ac2b-49cd-b9c9-459465d3d912\") " pod="openshift-image-registry/node-ca-t75rf"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-node-log\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640557    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1753a297-ac2b-49cd-b9c9-459465d3d912-host\") pod \"node-ca-t75rf\" (UID: \"1753a297-ac2b-49cd-b9c9-459465d3d912\") " pod="openshift-image-registry/node-ca-t75rf"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640577    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640603    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-sysctl-d\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640605    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-node-log\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.640625    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-run-ovn-kubernetes\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641138    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c58852df-cacb-4054-8f99-18344f10936d-konnectivity-ca\") pod \"konnectivity-agent-jnb9f\" (UID: \"c58852df-cacb-4054-8f99-18344f10936d\") " pod="kube-system/konnectivity-agent-jnb9f"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641201    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-var-lib-cni-multus\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641275    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-sysconfig\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641301    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-kubernetes\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641337    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-device-dir\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.641629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641373    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbqc9\" (UniqueName: \"kubernetes.io/projected/1753a297-ac2b-49cd-b9c9-459465d3d912-kube-api-access-rbqc9\") pod \"node-ca-t75rf\" (UID: \"1753a297-ac2b-49cd-b9c9-459465d3d912\") " pod="openshift-image-registry/node-ca-t75rf"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641408    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xrkx\" (UniqueName: \"kubernetes.io/projected/c717df31-e04f-4681-8af3-3c322acadca5-kube-api-access-7xrkx\") pod \"iptables-alerter-kdnwx\" (UID: \"c717df31-e04f-4681-8af3-3c322acadca5\") " pod="openshift-network-operator/iptables-alerter-kdnwx"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641601    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4099f20-afa3-482b-a22e-c4d59554f7fd-hosts-file\") pod \"node-resolver-lpvdl\" (UID: \"d4099f20-afa3-482b-a22e-c4d59554f7fd\") " pod="openshift-dns/node-resolver-lpvdl"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641632    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641706    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-device-dir\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641787    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-sys-fs\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641825    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-cni-bin\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641858    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b505067-ec95-4acc-bc12-554d0512edb2-env-overrides\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641859    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-run-ovn-kubernetes\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641918    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b505067-ec95-4acc-bc12-554d0512edb2-ovnkube-script-lib\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641953    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-multus-socket-dir-parent\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.641986    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5ae7904-5d7e-48dc-bb78-af622de153e4-multus-daemon-config\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.642212    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-system-cni-dir\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.642246    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h6vv\" (UniqueName: \"kubernetes.io/projected/a83d9228-1114-4294-b196-63f6bd1a0fdd-kube-api-access-7h6vv\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.642251    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-cni-bin\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.642313    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-run-multus-certs\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.642361    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-sys-fs\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.642459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.642433    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-kubelet\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.642998 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.642524    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-multus-cni-dir\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:52.642998 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.642567    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4099f20-afa3-482b-a22e-c4d59554f7fd-tmp-dir\") pod \"node-resolver-lpvdl\" (UID: \"d4099f20-afa3-482b-a22e-c4d59554f7fd\") " pod="openshift-dns/node-resolver-lpvdl"
Sep 09 18:31:52.642998 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.642607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-systemd\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.642998 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.642858    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b505067-ec95-4acc-bc12-554d0512edb2-host-kubelet\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.642998 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.642865    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h46xw\" (UniqueName: \"kubernetes.io/projected/66bae284-0ba9-48e6-af3e-7deafe38260e-kube-api-access-h46xw\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:52.642998 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.642973    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b505067-ec95-4acc-bc12-554d0512edb2-env-overrides\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.643641 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.643610    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b505067-ec95-4acc-bc12-554d0512edb2-ovnkube-script-lib\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.647251 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.647227    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b505067-ec95-4acc-bc12-554d0512edb2-ovn-node-metrics-cert\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.649512 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.649485    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf7kr\" (UniqueName: \"kubernetes.io/projected/2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a-kube-api-access-qf7kr\") pod \"aws-ebs-csi-driver-node-mxktw\" (UID: \"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:52.656540 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.656479    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7csw\" (UniqueName: \"kubernetes.io/projected/6b505067-ec95-4acc-bc12-554d0512edb2-kube-api-access-g7csw\") pod \"ovnkube-node-srx2c\" (UID: \"6b505067-ec95-4acc-bc12-554d0512edb2\") " pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:52.656540 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.656527    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbqc9\" (UniqueName: \"kubernetes.io/projected/1753a297-ac2b-49cd-b9c9-459465d3d912-kube-api-access-rbqc9\") pod \"node-ca-t75rf\" (UID: \"1753a297-ac2b-49cd-b9c9-459465d3d912\") " pod="openshift-image-registry/node-ca-t75rf"
Sep 09 18:31:52.672309 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.672236    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-63.ec2.internal" event={"ID":"b14058b333631407a76dac7ca4cc84a1","Type":"ContainerStarted","Data":"b1eb26ec088658ad5e842891b5fe5b95ea8f81440be50a5faf2a3087c09d794f"}
Sep 09 18:31:52.673855 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:52.673783    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal" event={"ID":"763960a7b123d75ea1d5a1fad095cc1b","Type":"ContainerStarted","Data":"b3741025a1d5c8f0c4608bc517aee8b69be65ad211f406a51a27103623da0d04"}
Sep 09 18:31:54.391395 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c717df31-e04f-4681-8af3-3c322acadca5-iptables-alerter-script\") pod \"iptables-alerter-kdnwx\" (UID: \"c717df31-e04f-4681-8af3-3c322acadca5\") " pod="openshift-network-operator/iptables-alerter-kdnwx"
Sep 09 18:31:54.391601 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391408    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-run-netns\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.391601 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.391601 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391465    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-run\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.391601 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391548    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-run\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.391601 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391550    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-run-netns\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.391829 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391603    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c58852df-cacb-4054-8f99-18344f10936d-agent-certs\") pod \"konnectivity-agent-jnb9f\" (UID: \"c58852df-cacb-4054-8f99-18344f10936d\") " pod="kube-system/konnectivity-agent-jnb9f"
Sep 09 18:31:54.391829 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66bae284-0ba9-48e6-af3e-7deafe38260e-tmp\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.391829 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391664    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.391829 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391700    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-sysctl-d\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.391829 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c58852df-cacb-4054-8f99-18344f10936d-konnectivity-ca\") pod \"konnectivity-agent-jnb9f\" (UID: \"c58852df-cacb-4054-8f99-18344f10936d\") " pod="kube-system/konnectivity-agent-jnb9f"
Sep 09 18:31:54.391829 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391749    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-var-lib-cni-multus\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.391829 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391774    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:31:54.391829 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-sysconfig\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.391829 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-kubernetes\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.392224 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391848    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xrkx\" (UniqueName: \"kubernetes.io/projected/c717df31-e04f-4681-8af3-3c322acadca5-kube-api-access-7xrkx\") pod \"iptables-alerter-kdnwx\" (UID: \"c717df31-e04f-4681-8af3-3c322acadca5\") " pod="openshift-network-operator/iptables-alerter-kdnwx"
Sep 09 18:31:54.392224 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391870    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4099f20-afa3-482b-a22e-c4d59554f7fd-hosts-file\") pod \"node-resolver-lpvdl\" (UID: \"d4099f20-afa3-482b-a22e-c4d59554f7fd\") " pod="openshift-dns/node-resolver-lpvdl"
Sep 09 18:31:54.392224 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391896    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-multus-socket-dir-parent\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.392224 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391920    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5ae7904-5d7e-48dc-bb78-af622de153e4-multus-daemon-config\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.392224 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391943    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-system-cni-dir\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.392224 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391964    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-sysctl-d\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.392519 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392307    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-var-lib-cni-multus\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.392519 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392310    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.392519 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:54.392417    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:54.392519 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:54.392474    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs podName:a83d9228-1114-4294-b196-63f6bd1a0fdd nodeName:}" failed. No retries permitted until 2026-09-09 18:31:54.892453721 +0000 UTC m=+3.129823528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs") pod "network-metrics-daemon-kdnq9" (UID: "a83d9228-1114-4294-b196-63f6bd1a0fdd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:54.392519 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.392761 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.391968    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h6vv\" (UniqueName: \"kubernetes.io/projected/a83d9228-1114-4294-b196-63f6bd1a0fdd-kube-api-access-7h6vv\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:31:54.392761 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392548    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c717df31-e04f-4681-8af3-3c322acadca5-iptables-alerter-script\") pod \"iptables-alerter-kdnwx\" (UID: \"c717df31-e04f-4681-8af3-3c322acadca5\") " pod="openshift-network-operator/iptables-alerter-kdnwx"
Sep 09 18:31:54.392761 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392571    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-run-multus-certs\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.392761 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-multus-cni-dir\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.392761 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392628    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4099f20-afa3-482b-a22e-c4d59554f7fd-tmp-dir\") pod \"node-resolver-lpvdl\" (UID: \"d4099f20-afa3-482b-a22e-c4d59554f7fd\") " pod="openshift-dns/node-resolver-lpvdl"
Sep 09 18:31:54.392761 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392652    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-systemd\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.392761 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392676    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h46xw\" (UniqueName: \"kubernetes.io/projected/66bae284-0ba9-48e6-af3e-7deafe38260e-kube-api-access-h46xw\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.392761 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392715    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-sysconfig\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.392761 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392759    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c717df31-e04f-4681-8af3-3c322acadca5-host-slash\") pod \"iptables-alerter-kdnwx\" (UID: \"c717df31-e04f-4681-8af3-3c322acadca5\") " pod="openshift-network-operator/iptables-alerter-kdnwx"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392769    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-kubernetes\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-cnibin\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392834    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-run-k8s-cni-cncf-io\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392845    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-cnibin\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392860    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-var-lib-kubelet\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392899    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5ae7904-5d7e-48dc-bb78-af622de153e4-cni-binary-copy\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392908    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-var-lib-kubelet\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392924    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-tuned\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392943    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tbfw\" (UniqueName: \"kubernetes.io/projected/d5ae7904-5d7e-48dc-bb78-af622de153e4-kube-api-access-5tbfw\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392963    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-multus-cni-dir\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392987    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-modprobe-d\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393002    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-sys\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393009    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-systemd\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393016    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-var-lib-kubelet\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-var-lib-cni-bin\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-cni-binary-copy\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393076    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcd4d\" (UniqueName: \"kubernetes.io/projected/d4099f20-afa3-482b-a22e-c4d59554f7fd-kube-api-access-qcd4d\") pod \"node-resolver-lpvdl\" (UID: \"d4099f20-afa3-482b-a22e-c4d59554f7fd\") " pod="openshift-dns/node-resolver-lpvdl"
Sep 09 18:31:54.393207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393098    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-cnibin\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393114    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-host\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393121    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c58852df-cacb-4054-8f99-18344f10936d-konnectivity-ca\") pod \"konnectivity-agent-jnb9f\" (UID: \"c58852df-cacb-4054-8f99-18344f10936d\") " pod="kube-system/konnectivity-agent-jnb9f"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393128    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-etc-kubernetes\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393153    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-etc-kubernetes\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393198    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-system-cni-dir\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-multus-conf-dir\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393246    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4099f20-afa3-482b-a22e-c4d59554f7fd-tmp-dir\") pod \"node-resolver-lpvdl\" (UID: \"d4099f20-afa3-482b-a22e-c4d59554f7fd\") " pod="openshift-dns/node-resolver-lpvdl"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393255    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbhg5\" (UniqueName: \"kubernetes.io/projected/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-kube-api-access-jbhg5\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393285    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpht\" (UniqueName: \"kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht\") pod \"network-check-target-fs6d5\" (UID: \"e27817cd-3adb-43ef-8434-7d43af58c903\") " pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393310    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-hostroot\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393326    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-run-multus-certs\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393336    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393362    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-sysctl-conf\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393385    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-lib-modules\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393464    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c717df31-e04f-4681-8af3-3c322acadca5-host-slash\") pod \"iptables-alerter-kdnwx\" (UID: \"c717df31-e04f-4681-8af3-3c322acadca5\") " pod="openshift-network-operator/iptables-alerter-kdnwx"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393494    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-var-lib-kubelet\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393535    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-system-cni-dir\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.394027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393591    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-sys\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393635    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-modprobe-d\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393696    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5ae7904-5d7e-48dc-bb78-af622de153e4-cni-binary-copy\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393764    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-cnibin\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393773    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-multus-conf-dir\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.392922    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-run-k8s-cni-cncf-io\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393818    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-host\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393846    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-multus-socket-dir-parent\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393854    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-host-var-lib-cni-bin\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393944    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d4099f20-afa3-482b-a22e-c4d59554f7fd-hosts-file\") pod \"node-resolver-lpvdl\" (UID: \"d4099f20-afa3-482b-a22e-c4d59554f7fd\") " pod="openshift-dns/node-resolver-lpvdl"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.393990    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-system-cni-dir\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.394045    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5ae7904-5d7e-48dc-bb78-af622de153e4-hostroot\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.394185    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-sysctl-conf\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.394262    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66bae284-0ba9-48e6-af3e-7deafe38260e-lib-modules\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.394323    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5ae7904-5d7e-48dc-bb78-af622de153e4-multus-daemon-config\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.394364    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-cni-binary-copy\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.394884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.394637    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.396072 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.395915    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66bae284-0ba9-48e6-af3e-7deafe38260e-tmp\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.396072 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.395945    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66bae284-0ba9-48e6-af3e-7deafe38260e-etc-tuned\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.396427 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.396406    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c58852df-cacb-4054-8f99-18344f10936d-agent-certs\") pod \"konnectivity-agent-jnb9f\" (UID: \"c58852df-cacb-4054-8f99-18344f10936d\") " pod="kube-system/konnectivity-agent-jnb9f"
Sep 09 18:31:54.399744 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:54.399721    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 18:31:54.399842 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:54.399752    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 18:31:54.399842 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:54.399765    2575 projected.go:196] Error preparing data for projected volume kube-api-access-zzpht for pod openshift-network-diagnostics/network-check-target-fs6d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:54.399999 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:54.399842    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht podName:e27817cd-3adb-43ef-8434-7d43af58c903 nodeName:}" failed. No retries permitted until 2026-09-09 18:31:54.899824345 +0000 UTC m=+3.137194142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zzpht" (UniqueName: "kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht") pod "network-check-target-fs6d5" (UID: "e27817cd-3adb-43ef-8434-7d43af58c903") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:54.403615 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.403067    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xrkx\" (UniqueName: \"kubernetes.io/projected/c717df31-e04f-4681-8af3-3c322acadca5-kube-api-access-7xrkx\") pod \"iptables-alerter-kdnwx\" (UID: \"c717df31-e04f-4681-8af3-3c322acadca5\") " pod="openshift-network-operator/iptables-alerter-kdnwx"
Sep 09 18:31:54.403615 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.403564    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tbfw\" (UniqueName: \"kubernetes.io/projected/d5ae7904-5d7e-48dc-bb78-af622de153e4-kube-api-access-5tbfw\") pod \"multus-qdrf9\" (UID: \"d5ae7904-5d7e-48dc-bb78-af622de153e4\") " pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.403615 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.403580    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h46xw\" (UniqueName: \"kubernetes.io/projected/66bae284-0ba9-48e6-af3e-7deafe38260e-kube-api-access-h46xw\") pod \"tuned-jtwwp\" (UID: \"66bae284-0ba9-48e6-af3e-7deafe38260e\") " pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.403981 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.403939    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcd4d\" (UniqueName: \"kubernetes.io/projected/d4099f20-afa3-482b-a22e-c4d59554f7fd-kube-api-access-qcd4d\") pod \"node-resolver-lpvdl\" (UID: \"d4099f20-afa3-482b-a22e-c4d59554f7fd\") " pod="openshift-dns/node-resolver-lpvdl"
Sep 09 18:31:54.404313 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.404289    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h6vv\" (UniqueName: \"kubernetes.io/projected/a83d9228-1114-4294-b196-63f6bd1a0fdd-kube-api-access-7h6vv\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:31:54.404883 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.404826    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbhg5\" (UniqueName: \"kubernetes.io/projected/2bd43ab1-614e-4ef0-b133-4d5e4f6d949b-kube-api-access-jbhg5\") pod \"multus-additional-cni-plugins-pmjtg\" (UID: \"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b\") " pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.478266 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.478227    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t75rf"
Sep 09 18:31:54.486529 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.486485    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw"
Sep 09 18:31:54.496417 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.496392    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:31:54.503260 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.503227    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kdnwx"
Sep 09 18:31:54.510069 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.510041    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jtwwp"
Sep 09 18:31:54.515929 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.515907    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qdrf9"
Sep 09 18:31:54.524873 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.524845    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pmjtg"
Sep 09 18:31:54.532654 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.532625    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jnb9f"
Sep 09 18:31:54.537709 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.537682    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lpvdl"
Sep 09 18:31:54.796467 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.796377    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:31:54.796635 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:54.796557    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:54.796701 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:54.796637    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret podName:3526bb20-1563-4c64-970b-e159d015bd3d nodeName:}" failed. No retries permitted until 2026-09-09 18:31:55.796614472 +0000 UTC m=+4.033984276 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret") pod "global-pull-secret-syncer-7vhcs" (UID: "3526bb20-1563-4c64-970b-e159d015bd3d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:54.897742 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.897701    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:31:54.897933 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:54.897884    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:54.897995 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:54.897969    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs podName:a83d9228-1114-4294-b196-63f6bd1a0fdd nodeName:}" failed. No retries permitted until 2026-09-09 18:31:55.897945499 +0000 UTC m=+4.135315306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs") pod "network-metrics-daemon-kdnq9" (UID: "a83d9228-1114-4294-b196-63f6bd1a0fdd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:54.998635 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:54.998600    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpht\" (UniqueName: \"kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht\") pod \"network-check-target-fs6d5\" (UID: \"e27817cd-3adb-43ef-8434-7d43af58c903\") " pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:31:54.998825 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:54.998801    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 18:31:54.998884 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:54.998830    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 18:31:54.998884 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:54.998844    2575 projected.go:196] Error preparing data for projected volume kube-api-access-zzpht for pod openshift-network-diagnostics/network-check-target-fs6d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:54.998976 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:54.998916    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht podName:e27817cd-3adb-43ef-8434-7d43af58c903 nodeName:}" failed. No retries permitted until 2026-09-09 18:31:55.998895092 +0000 UTC m=+4.236264904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zzpht" (UniqueName: "kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht") pod "network-check-target-fs6d5" (UID: "e27817cd-3adb-43ef-8434-7d43af58c903") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:55.218584 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:55.218476    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 18:26:54 +0000 UTC" deadline="2028-02-14 06:33:35.132956542 +0000 UTC"
Sep 09 18:31:55.218584 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:55.218526    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12540h1m39.914434912s"
Sep 09 18:31:55.526259 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:55.526227    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ae7904_5d7e_48dc_bb78_af622de153e4.slice/crio-1e57f67618ccf054e21ae59de2380ba1e31f34689830077b2df516b8936dbcf3 WatchSource:0}: Error finding container 1e57f67618ccf054e21ae59de2380ba1e31f34689830077b2df516b8936dbcf3: Status 404 returned error can't find the container with id 1e57f67618ccf054e21ae59de2380ba1e31f34689830077b2df516b8936dbcf3
Sep 09 18:31:55.528072 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:55.528020    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b505067_ec95_4acc_bc12_554d0512edb2.slice/crio-c2c5430d77f99c03a40ee411472f3c7d3e3685a55642309915988f64fc2dd81e WatchSource:0}: Error finding container c2c5430d77f99c03a40ee411472f3c7d3e3685a55642309915988f64fc2dd81e: Status 404 returned error can't find the container with id c2c5430d77f99c03a40ee411472f3c7d3e3685a55642309915988f64fc2dd81e
Sep 09 18:31:55.532051 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:55.532018    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58852df_cacb_4054_8f99_18344f10936d.slice/crio-f3cc2720995a5efebc0025cef2074b7266be221ae93bae4905398efa9555253c WatchSource:0}: Error finding container f3cc2720995a5efebc0025cef2074b7266be221ae93bae4905398efa9555253c: Status 404 returned error can't find the container with id f3cc2720995a5efebc0025cef2074b7266be221ae93bae4905398efa9555253c
Sep 09 18:31:55.532827 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:55.532801    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd43ab1_614e_4ef0_b133_4d5e4f6d949b.slice/crio-7057c5dded6131d3bcfbcc9977c63f219ea940d39035b0643725c23038360233 WatchSource:0}: Error finding container 7057c5dded6131d3bcfbcc9977c63f219ea940d39035b0643725c23038360233: Status 404 returned error can't find the container with id 7057c5dded6131d3bcfbcc9977c63f219ea940d39035b0643725c23038360233
Sep 09 18:31:55.533627 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:55.533484    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfcc95e_6e5a_438c_b8bc_891d6e8dcb4a.slice/crio-0d59d5874b4bf64abb445c0c2ee3cded6a716065f0fa33b1819b7a34f942c05e WatchSource:0}: Error finding container 0d59d5874b4bf64abb445c0c2ee3cded6a716065f0fa33b1819b7a34f942c05e: Status 404 returned error can't find the container with id 0d59d5874b4bf64abb445c0c2ee3cded6a716065f0fa33b1819b7a34f942c05e
Sep 09 18:31:55.534912 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:55.534883    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1753a297_ac2b_49cd_b9c9_459465d3d912.slice/crio-09d1572132a2369201047305ff7f496cc137bbf14e95d8cbcbdb35147423d503 WatchSource:0}: Error finding container 09d1572132a2369201047305ff7f496cc137bbf14e95d8cbcbdb35147423d503: Status 404 returned error can't find the container with id 09d1572132a2369201047305ff7f496cc137bbf14e95d8cbcbdb35147423d503
Sep 09 18:31:55.535788 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:55.535769    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bae284_0ba9_48e6_af3e_7deafe38260e.slice/crio-03aac304513b5bc88cc4fdc4ca964f9b59c2295c10d4040b7e8c747eeeba36f0 WatchSource:0}: Error finding container 03aac304513b5bc88cc4fdc4ca964f9b59c2295c10d4040b7e8c747eeeba36f0: Status 404 returned error can't find the container with id 03aac304513b5bc88cc4fdc4ca964f9b59c2295c10d4040b7e8c747eeeba36f0
Sep 09 18:31:55.537286 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:31:55.537261    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4099f20_afa3_482b_a22e_c4d59554f7fd.slice/crio-07a8b14ed516b97af3fa03b9f91bbf3a5af7da8485e0aa1c5516824788f8b042 WatchSource:0}: Error finding container 07a8b14ed516b97af3fa03b9f91bbf3a5af7da8485e0aa1c5516824788f8b042: Status 404 returned error can't find the container with id 07a8b14ed516b97af3fa03b9f91bbf3a5af7da8485e0aa1c5516824788f8b042
Sep 09 18:31:55.805328 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:55.805293    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:31:55.805472 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:55.805440    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:55.805547 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:55.805521    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret podName:3526bb20-1563-4c64-970b-e159d015bd3d nodeName:}" failed. No retries permitted until 2026-09-09 18:31:57.805490203 +0000 UTC m=+6.042860000 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret") pod "global-pull-secret-syncer-7vhcs" (UID: "3526bb20-1563-4c64-970b-e159d015bd3d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:55.906368 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:55.906323    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:31:55.906558 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:55.906464    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:55.906605 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:55.906561    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs podName:a83d9228-1114-4294-b196-63f6bd1a0fdd nodeName:}" failed. No retries permitted until 2026-09-09 18:31:57.906545259 +0000 UTC m=+6.143915052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs") pod "network-metrics-daemon-kdnq9" (UID: "a83d9228-1114-4294-b196-63f6bd1a0fdd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:56.006715 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.006675    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpht\" (UniqueName: \"kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht\") pod \"network-check-target-fs6d5\" (UID: \"e27817cd-3adb-43ef-8434-7d43af58c903\") " pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:31:56.006907 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:56.006842    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 18:31:56.006907 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:56.006869    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 18:31:56.006907 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:56.006883    2575 projected.go:196] Error preparing data for projected volume kube-api-access-zzpht for pod openshift-network-diagnostics/network-check-target-fs6d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:56.007077 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:56.006991    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht podName:e27817cd-3adb-43ef-8434-7d43af58c903 nodeName:}" failed. No retries permitted until 2026-09-09 18:31:58.006969569 +0000 UTC m=+6.244339385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zzpht" (UniqueName: "kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht") pod "network-check-target-fs6d5" (UID: "e27817cd-3adb-43ef-8434-7d43af58c903") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:56.218949 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.218876    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 18:26:54 +0000 UTC" deadline="2028-04-14 03:59:16.704521562 +0000 UTC"
Sep 09 18:31:56.218949 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.218948    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13977h27m20.485578456s"
Sep 09 18:31:56.317641 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.317607    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:31:56.317827 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:56.317755    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:31:56.318489 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.318170    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:31:56.318489 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:56.318302    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:31:56.318489 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.318363    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:31:56.318489 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:56.318428    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:31:56.330203 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.329017    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-63.ec2.internal" event={"ID":"b14058b333631407a76dac7ca4cc84a1","Type":"ContainerStarted","Data":"a40796dcf49b1aca7211abe8837f8e535e4824e5fb6c334da4bc0662014a9d87"}
Sep 09 18:31:56.337356 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.337259    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lpvdl" event={"ID":"d4099f20-afa3-482b-a22e-c4d59554f7fd","Type":"ContainerStarted","Data":"07a8b14ed516b97af3fa03b9f91bbf3a5af7da8485e0aa1c5516824788f8b042"}
Sep 09 18:31:56.340403 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.340316    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t75rf" event={"ID":"1753a297-ac2b-49cd-b9c9-459465d3d912","Type":"ContainerStarted","Data":"09d1572132a2369201047305ff7f496cc137bbf14e95d8cbcbdb35147423d503"}
Sep 09 18:31:56.344093 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.343993    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmjtg" event={"ID":"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b","Type":"ContainerStarted","Data":"7057c5dded6131d3bcfbcc9977c63f219ea940d39035b0643725c23038360233"}
Sep 09 18:31:56.351838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.351773    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw" event={"ID":"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a","Type":"ContainerStarted","Data":"0d59d5874b4bf64abb445c0c2ee3cded6a716065f0fa33b1819b7a34f942c05e"}
Sep 09 18:31:56.353673 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.353632    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jnb9f" event={"ID":"c58852df-cacb-4054-8f99-18344f10936d","Type":"ContainerStarted","Data":"f3cc2720995a5efebc0025cef2074b7266be221ae93bae4905398efa9555253c"}
Sep 09 18:31:56.357978 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.357855    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c" event={"ID":"6b505067-ec95-4acc-bc12-554d0512edb2","Type":"ContainerStarted","Data":"c2c5430d77f99c03a40ee411472f3c7d3e3685a55642309915988f64fc2dd81e"}
Sep 09 18:31:56.361137 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.360981    2575 generic.go:356] "Generic (PLEG): container finished" podID="763960a7b123d75ea1d5a1fad095cc1b" containerID="7bf89df5dcc8b49a7fd48b6312da9768c9bce2681757e986e710d443803306a5" exitCode=0
Sep 09 18:31:56.361137 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.361077    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal" event={"ID":"763960a7b123d75ea1d5a1fad095cc1b","Type":"ContainerDied","Data":"7bf89df5dcc8b49a7fd48b6312da9768c9bce2681757e986e710d443803306a5"}
Sep 09 18:31:56.364638 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.364605    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kdnwx" event={"ID":"c717df31-e04f-4681-8af3-3c322acadca5","Type":"ContainerStarted","Data":"fc20dc084afc3fe7db055d1029f39e92fe6bc974283f03ea3172871a4c385578"}
Sep 09 18:31:56.367534 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.367490    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jtwwp" event={"ID":"66bae284-0ba9-48e6-af3e-7deafe38260e","Type":"ContainerStarted","Data":"03aac304513b5bc88cc4fdc4ca964f9b59c2295c10d4040b7e8c747eeeba36f0"}
Sep 09 18:31:56.369239 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.369211    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qdrf9" event={"ID":"d5ae7904-5d7e-48dc-bb78-af622de153e4","Type":"ContainerStarted","Data":"1e57f67618ccf054e21ae59de2380ba1e31f34689830077b2df516b8936dbcf3"}
Sep 09 18:31:56.379841 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:56.379486    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-63.ec2.internal" podStartSLOduration=3.37946563 podStartE2EDuration="3.37946563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:31:56.343046545 +0000 UTC m=+4.580416362" watchObservedRunningTime="2026-09-09 18:31:56.37946563 +0000 UTC m=+4.616835446"
Sep 09 18:31:57.388673 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:57.388626    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal" event={"ID":"763960a7b123d75ea1d5a1fad095cc1b","Type":"ContainerStarted","Data":"bac7c0b661ef70ab7e1e041ee116ebeef2d3a5ea1d85e3e2d6f06e2cd740217f"}
Sep 09 18:31:57.406949 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:57.406591    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-63.ec2.internal" podStartSLOduration=4.406567652 podStartE2EDuration="4.406567652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:31:57.405608564 +0000 UTC m=+5.642978392" watchObservedRunningTime="2026-09-09 18:31:57.406567652 +0000 UTC m=+5.643937468"
Sep 09 18:31:57.821702 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:57.821609    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:31:57.821864 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:57.821816    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:57.821934 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:57.821884    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret podName:3526bb20-1563-4c64-970b-e159d015bd3d nodeName:}" failed. No retries permitted until 2026-09-09 18:32:01.821865905 +0000 UTC m=+10.059235714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret") pod "global-pull-secret-syncer-7vhcs" (UID: "3526bb20-1563-4c64-970b-e159d015bd3d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 18:31:57.922795 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:57.922758    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:31:57.922989 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:57.922955    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:57.923051 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:57.923034    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs podName:a83d9228-1114-4294-b196-63f6bd1a0fdd nodeName:}" failed. No retries permitted until 2026-09-09 18:32:01.923014808 +0000 UTC m=+10.160384617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs") pod "network-metrics-daemon-kdnq9" (UID: "a83d9228-1114-4294-b196-63f6bd1a0fdd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:31:58.023469 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:58.023416    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpht\" (UniqueName: \"kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht\") pod \"network-check-target-fs6d5\" (UID: \"e27817cd-3adb-43ef-8434-7d43af58c903\") " pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:31:58.023715 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:58.023693    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 18:31:58.023778 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:58.023723    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 18:31:58.023778 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:58.023737    2575 projected.go:196] Error preparing data for projected volume kube-api-access-zzpht for pod openshift-network-diagnostics/network-check-target-fs6d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:58.023879 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:58.023809    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht podName:e27817cd-3adb-43ef-8434-7d43af58c903 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:02.023788387 +0000 UTC m=+10.261158186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zzpht" (UniqueName: "kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht") pod "network-check-target-fs6d5" (UID: "e27817cd-3adb-43ef-8434-7d43af58c903") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:31:58.319622 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:58.319530    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:31:58.319796 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:58.319714    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:31:58.320090 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:58.320055    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:31:58.320309 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:58.320228    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:31:58.321663 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:31:58.320582    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:31:58.321663 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:31:58.320704    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:00.315561 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:00.314756    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:00.315561 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:00.314896    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:00.315561 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:00.315292    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:00.315561 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:00.315397    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:00.315561 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:00.315443    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:00.315561 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:00.315521    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:01.853329 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:01.853090    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:01.853791 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:01.853294    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 18:32:01.853791 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:01.853431    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret podName:3526bb20-1563-4c64-970b-e159d015bd3d nodeName:}" failed. No retries permitted until 2026-09-09 18:32:09.853414613 +0000 UTC m=+18.090784426 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret") pod "global-pull-secret-syncer-7vhcs" (UID: "3526bb20-1563-4c64-970b-e159d015bd3d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 18:32:01.954573 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:01.954530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:01.954751 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:01.954686    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:32:01.954808 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:01.954769    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs podName:a83d9228-1114-4294-b196-63f6bd1a0fdd nodeName:}" failed. No retries permitted until 2026-09-09 18:32:09.95474781 +0000 UTC m=+18.192117618 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs") pod "network-metrics-daemon-kdnq9" (UID: "a83d9228-1114-4294-b196-63f6bd1a0fdd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:32:02.055923 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:02.055846    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpht\" (UniqueName: \"kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht\") pod \"network-check-target-fs6d5\" (UID: \"e27817cd-3adb-43ef-8434-7d43af58c903\") " pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:02.056122 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:02.056003    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 18:32:02.056122 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:02.056030    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 18:32:02.056122 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:02.056041    2575 projected.go:196] Error preparing data for projected volume kube-api-access-zzpht for pod openshift-network-diagnostics/network-check-target-fs6d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:32:02.056122 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:02.056109    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht podName:e27817cd-3adb-43ef-8434-7d43af58c903 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:10.056094652 +0000 UTC m=+18.293464445 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zzpht" (UniqueName: "kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht") pod "network-check-target-fs6d5" (UID: "e27817cd-3adb-43ef-8434-7d43af58c903") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:32:02.315612 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:02.315480    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:02.315793 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:02.315622    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:02.315793 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:02.315638    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:02.315793 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:02.315769    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:02.315941 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:02.315799    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:02.315941 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:02.315839    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:04.314918 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.314885    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:04.315431 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.314900    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:04.315431 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:04.315011    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:04.315431 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:04.315088    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:04.315431 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.314904    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:04.315431 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:04.315162    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:04.405529 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.405472    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lpvdl" event={"ID":"d4099f20-afa3-482b-a22e-c4d59554f7fd","Type":"ContainerStarted","Data":"0e4af695818c2d7e88efb5db4f95c9a7396651a5fe143ad578f58790e4d956fc"}
Sep 09 18:32:04.408415 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.408343    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t75rf" event={"ID":"1753a297-ac2b-49cd-b9c9-459465d3d912","Type":"ContainerStarted","Data":"7c4729ebadcd4ea7f41f3213490de12bc4c81cd5406ed424f3877389dec719ee"}
Sep 09 18:32:04.411186 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.411153    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmjtg" event={"ID":"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b","Type":"ContainerStarted","Data":"95f43b1dde778aea8e5ab9a4a5a0230bcb279fdf15cd0b05cf22803ee3af05a8"}
Sep 09 18:32:04.414229 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.413910    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw" event={"ID":"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a","Type":"ContainerStarted","Data":"9ba61188212b48499274def5046894601c0fe114aef12b8d6251324b0e06920d"}
Sep 09 18:32:04.418053 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.417307    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jnb9f" event={"ID":"c58852df-cacb-4054-8f99-18344f10936d","Type":"ContainerStarted","Data":"dba24eb103b3c5c9ad4b324b600e28d0a87d90449cce317e928be94f9d4819ac"}
Sep 09 18:32:04.420602 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.420157    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lpvdl" podStartSLOduration=4.481983063 podStartE2EDuration="12.420132453s" totalImagesPullingTime="7.93814939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:04.419735448 +0000 UTC m=+12.657105265" watchObservedRunningTime="2026-09-09 18:32:04.420132453 +0000 UTC m=+12.657502268"
Sep 09 18:32:04.423876 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.423841    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jtwwp" event={"ID":"66bae284-0ba9-48e6-af3e-7deafe38260e","Type":"ContainerStarted","Data":"a5500fc8d285eab86d0217a1204a0d0a2e24e7634008dfc74b370d20fa72b884"}
Sep 09 18:32:04.458184 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.458107    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t75rf" podStartSLOduration=4.518475881 podStartE2EDuration="12.458081608s" totalImagesPullingTime="7.939605727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:04.434330316 +0000 UTC m=+12.671700172" watchObservedRunningTime="2026-09-09 18:32:04.458081608 +0000 UTC m=+12.695451426"
Sep 09 18:32:04.463032 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.462942    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jnb9f"
Sep 09 18:32:04.463906 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.463821    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jnb9f"
Sep 09 18:32:04.489639 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.488658    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jnb9f" podStartSLOduration=4.548098442 podStartE2EDuration="12.488635995s" totalImagesPullingTime="7.940537553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:04.468862779 +0000 UTC m=+12.706232594" watchObservedRunningTime="2026-09-09 18:32:04.488635995 +0000 UTC m=+12.726005813"
Sep 09 18:32:04.505340 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:04.505272    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jtwwp" podStartSLOduration=4.541834615 podStartE2EDuration="12.505245766s" totalImagesPullingTime="7.963411151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:04.5043229 +0000 UTC m=+12.741692717" watchObservedRunningTime="2026-09-09 18:32:04.505245766 +0000 UTC m=+12.742615611"
Sep 09 18:32:05.428615 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:05.428570    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kdnwx" event={"ID":"c717df31-e04f-4681-8af3-3c322acadca5","Type":"ContainerStarted","Data":"98cd3282257def72f6570353d4773d3243782c4ef7ae64a2b37c7ca120e6a637"}
Sep 09 18:32:05.429979 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:05.429945    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jnb9f"
Sep 09 18:32:05.430105 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:05.430009    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jnb9f"
Sep 09 18:32:05.443624 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:05.443495    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kdnwx" podStartSLOduration=5.509626116 podStartE2EDuration="13.443470735s" totalImagesPullingTime="7.933844619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:05.442820333 +0000 UTC m=+13.680190150" watchObservedRunningTime="2026-09-09 18:32:05.443470735 +0000 UTC m=+13.680840552"
Sep 09 18:32:06.315208 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:06.315176    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:06.315396 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:06.315186    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:06.315396 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:06.315287    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:06.315396 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:06.315341    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:06.315609 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:06.315442    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:06.315609 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:06.315545    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:08.316192 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:08.316116    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:08.316554 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:08.316221    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:08.316633 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:08.316579    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:08.316705 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:08.316678    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:08.316833 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:08.316753    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:08.316896 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:08.316842    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:08.438387 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:08.438187    2575 generic.go:356] "Generic (PLEG): container finished" podID="2bd43ab1-614e-4ef0-b133-4d5e4f6d949b" containerID="95f43b1dde778aea8e5ab9a4a5a0230bcb279fdf15cd0b05cf22803ee3af05a8" exitCode=0
Sep 09 18:32:08.438568 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:08.438266    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmjtg" event={"ID":"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b","Type":"ContainerDied","Data":"95f43b1dde778aea8e5ab9a4a5a0230bcb279fdf15cd0b05cf22803ee3af05a8"}
Sep 09 18:32:09.616694 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:09.616663    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 18:32:09.920151 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:09.920102    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:09.920315 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:09.920280    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 18:32:09.920370 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:09.920359    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret podName:3526bb20-1563-4c64-970b-e159d015bd3d nodeName:}" failed. No retries permitted until 2026-09-09 18:32:25.920337363 +0000 UTC m=+34.157707173 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret") pod "global-pull-secret-syncer-7vhcs" (UID: "3526bb20-1563-4c64-970b-e159d015bd3d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 18:32:10.021517 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:10.021463    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:10.021704 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:10.021643    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:32:10.021776 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:10.021725    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs podName:a83d9228-1114-4294-b196-63f6bd1a0fdd nodeName:}" failed. No retries permitted until 2026-09-09 18:32:26.021702026 +0000 UTC m=+34.259071820 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs") pod "network-metrics-daemon-kdnq9" (UID: "a83d9228-1114-4294-b196-63f6bd1a0fdd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:32:10.125609 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:10.123332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpht\" (UniqueName: \"kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht\") pod \"network-check-target-fs6d5\" (UID: \"e27817cd-3adb-43ef-8434-7d43af58c903\") " pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:10.125609 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:10.123584    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 18:32:10.125609 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:10.123607    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 18:32:10.125609 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:10.123621    2575 projected.go:196] Error preparing data for projected volume kube-api-access-zzpht for pod openshift-network-diagnostics/network-check-target-fs6d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:32:10.125609 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:10.123704    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht podName:e27817cd-3adb-43ef-8434-7d43af58c903 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:26.123683725 +0000 UTC m=+34.361053541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zzpht" (UniqueName: "kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht") pod "network-check-target-fs6d5" (UID: "e27817cd-3adb-43ef-8434-7d43af58c903") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:32:10.248626 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:10.248482    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T18:32:09.616679133Z","UUID":"0c0758f5-d0d0-4c0f-9a53-efb534ec9b8e","Handler":null,"Name":"","Endpoint":""}
Sep 09 18:32:10.250464 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:10.250420    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 18:32:10.250651 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:10.250482    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 18:32:10.316383 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:10.316346    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:10.316563 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:10.316475    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:10.316907 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:10.316886    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:10.317010 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:10.316989    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:10.317074 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:10.317010    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:10.317116 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:10.317096    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:10.445149 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:10.445060    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qdrf9" event={"ID":"d5ae7904-5d7e-48dc-bb78-af622de153e4","Type":"ContainerStarted","Data":"55afe710001e845eb547b9d14007f1902a21bb170e50812741e5968aa69999c8"}
Sep 09 18:32:10.447437 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:10.447399    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw" event={"ID":"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a","Type":"ContainerStarted","Data":"9281b8c640df7fe189968de0476ae1b951450bc580cce6b78859cedbac785b2d"}
Sep 09 18:32:12.316040 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:12.316000    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:12.316646 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:12.316094    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:12.316646 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:12.316126    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:12.316646 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:12.316180    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:12.316646 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:12.316203    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:12.316646 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:12.316265    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:14.314556 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:14.314515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:14.315138 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:14.314515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:14.315138 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:14.314657    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:14.315138 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:14.314730    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:14.315138 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:14.314515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:14.315138 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:14.314837    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:16.315061 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:16.314874    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:16.315546 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:16.314874    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:16.315546 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:16.315151    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:16.315546 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:16.314874    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:16.315546 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:16.315257    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:16.315546 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:16.315309    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:16.462199 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:16.461958    2575 generic.go:356] "Generic (PLEG): container finished" podID="2bd43ab1-614e-4ef0-b133-4d5e4f6d949b" containerID="92c19399fed71fe2a3e2bb7c623dc1c21d88a9e64f3624e8490cef0d8c092615" exitCode=0
Sep 09 18:32:16.462330 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:16.462031    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmjtg" event={"ID":"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b","Type":"ContainerDied","Data":"92c19399fed71fe2a3e2bb7c623dc1c21d88a9e64f3624e8490cef0d8c092615"}
Sep 09 18:32:16.464988 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:16.464960    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw" event={"ID":"2dfcc95e-6e5a-438c-b8bc-891d6e8dcb4a","Type":"ContainerStarted","Data":"5b6b5b1c05c7963abe0585b966623ef8b83447d866945e612a2848777b1cabdc"}
Sep 09 18:32:16.467493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:16.467469    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c" event={"ID":"6b505067-ec95-4acc-bc12-554d0512edb2","Type":"ContainerStarted","Data":"16f146b36dc5f6e026914b4aff16c91808761a1dd28d593741a9413936bd3bc6"}
Sep 09 18:32:16.467610 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:16.467550    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c" event={"ID":"6b505067-ec95-4acc-bc12-554d0512edb2","Type":"ContainerStarted","Data":"dff777e01a33396dd3150116d0739e1ce45ab2f5c8126cbfa816e36453af0c8e"}
Sep 09 18:32:16.467610 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:16.467581    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c" event={"ID":"6b505067-ec95-4acc-bc12-554d0512edb2","Type":"ContainerStarted","Data":"c70a1b57dc927739e0e6a97bde1cbf5253090c2ea5436dcbcbbe6576b0bc016b"}
Sep 09 18:32:16.467691 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:16.467613    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c" event={"ID":"6b505067-ec95-4acc-bc12-554d0512edb2","Type":"ContainerStarted","Data":"8c8876ec5b14343595e449f27ab938b8dcfe951f33f6acb6622be9649dbb2230"}
Sep 09 18:32:16.484349 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:16.484296    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qdrf9" podStartSLOduration=10.506607846 podStartE2EDuration="24.484276437s" totalImagesPullingTime="13.977668591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:10.460864745 +0000 UTC m=+18.698234561" watchObservedRunningTime="2026-09-09 18:32:16.484276437 +0000 UTC m=+24.721646253"
Sep 09 18:32:16.499181 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:16.499139    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxktw" podStartSLOduration=4.168623066 podStartE2EDuration="24.499124039s" totalImagesPullingTime="20.330500973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:16.498975594 +0000 UTC m=+24.736345411" watchObservedRunningTime="2026-09-09 18:32:16.499124039 +0000 UTC m=+24.736493850"
Sep 09 18:32:17.472380 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:17.472294    2575 generic.go:356] "Generic (PLEG): container finished" podID="2bd43ab1-614e-4ef0-b133-4d5e4f6d949b" containerID="9cd833077ae6f6a6a8e15109eb0c8f33174ba7d9acbb5ffd7516cf6f365b7cb2" exitCode=0
Sep 09 18:32:17.473005 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:17.472372    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmjtg" event={"ID":"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b","Type":"ContainerDied","Data":"9cd833077ae6f6a6a8e15109eb0c8f33174ba7d9acbb5ffd7516cf6f365b7cb2"}
Sep 09 18:32:17.475902 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:17.475872    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c" event={"ID":"6b505067-ec95-4acc-bc12-554d0512edb2","Type":"ContainerStarted","Data":"2753128be7c5d9916d77f415079e729c31fdf096629dba70bc570cc782e67781"}
Sep 09 18:32:17.475999 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:17.475952    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c" event={"ID":"6b505067-ec95-4acc-bc12-554d0512edb2","Type":"ContainerStarted","Data":"cf28af8ce61ebcaccd38c4d545e48359ca6dd5647e45ccef9befd0671f672846"}
Sep 09 18:32:18.315314 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:18.315215    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:18.315314 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:18.315252    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:18.315314 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:18.315220    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:18.315586 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:18.315324    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:18.315586 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:18.315415    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:18.315586 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:18.315470    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:18.480512 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:18.480476    2575 generic.go:356] "Generic (PLEG): container finished" podID="2bd43ab1-614e-4ef0-b133-4d5e4f6d949b" containerID="540c0f391af0fcd4c559f89246663da582ae4d54787a3b845eec663daeb0d30b" exitCode=0
Sep 09 18:32:18.480915 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:18.480550    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmjtg" event={"ID":"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b","Type":"ContainerDied","Data":"540c0f391af0fcd4c559f89246663da582ae4d54787a3b845eec663daeb0d30b"}
Sep 09 18:32:19.486276 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:19.486241    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c" event={"ID":"6b505067-ec95-4acc-bc12-554d0512edb2","Type":"ContainerStarted","Data":"3b5d8bdcf28bef1c100242e0a9c9b8be1c9643ad95661c15053bf340bcf3668b"}
Sep 09 18:32:20.315144 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:20.315105    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:20.315144 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:20.315105    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:20.315365 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:20.315118    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:20.315365 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:20.315241    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:20.315365 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:20.315310    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:20.315462 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:20.315380    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:21.493845 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:21.493573    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c" event={"ID":"6b505067-ec95-4acc-bc12-554d0512edb2","Type":"ContainerStarted","Data":"aa97a237ffc9baaac65de51123c6865b6564df5182d1132237cb2276781b7f3d"}
Sep 09 18:32:21.494373 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:21.493901    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:32:21.494373 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:21.493960    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:32:21.510111 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:21.510084    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:32:21.523979 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:21.523929    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c" podStartSLOduration=9.189643042 podStartE2EDuration="29.523912861s" totalImagesPullingTime="20.334269819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:21.523166384 +0000 UTC m=+29.760536223" watchObservedRunningTime="2026-09-09 18:32:21.523912861 +0000 UTC m=+29.761282676"
Sep 09 18:32:22.316524 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:22.316230    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:22.316524 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:22.316323    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:22.316524 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:22.316342    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:22.316524 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:22.316386    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:22.316524 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:22.316447    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:22.316958 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:22.316536    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:22.496709 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:22.496675    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:32:22.514973 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:22.514940    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:32:22.622209 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:22.622083    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kdnq9"]
Sep 09 18:32:22.622374 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:22.622233    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:22.622374 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:22.622348    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:22.625376 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:22.625079    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7vhcs"]
Sep 09 18:32:22.625376 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:22.625199    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:22.625376 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:22.625320    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:22.625961 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:22.625932    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fs6d5"]
Sep 09 18:32:22.626080 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:22.626042    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:22.626145 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:22.626130    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:24.314625 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:24.314589    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:24.314625 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:24.314611    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:24.314625 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:24.314636    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:24.315181 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:24.314743    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:24.315181 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:24.314821    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:24.315181 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:24.314910    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:25.941684 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:25.941412    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:25.942073 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:25.941581    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 18:32:25.942073 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:25.941755    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret podName:3526bb20-1563-4c64-970b-e159d015bd3d nodeName:}" failed. No retries permitted until 2026-09-09 18:32:57.94173789 +0000 UTC m=+66.179107706 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret") pod "global-pull-secret-syncer-7vhcs" (UID: "3526bb20-1563-4c64-970b-e159d015bd3d") : object "kube-system"/"original-pull-secret" not registered
Sep 09 18:32:26.043000 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:26.042965    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:26.043186 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:26.043085    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:32:26.043186 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:26.043139    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs podName:a83d9228-1114-4294-b196-63f6bd1a0fdd nodeName:}" failed. No retries permitted until 2026-09-09 18:32:58.043124852 +0000 UTC m=+66.280494645 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs") pod "network-metrics-daemon-kdnq9" (UID: "a83d9228-1114-4294-b196-63f6bd1a0fdd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 18:32:26.143473 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:26.143434    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpht\" (UniqueName: \"kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht\") pod \"network-check-target-fs6d5\" (UID: \"e27817cd-3adb-43ef-8434-7d43af58c903\") " pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:26.143692 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:26.143635    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 18:32:26.143692 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:26.143668    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 18:32:26.143692 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:26.143681    2575 projected.go:196] Error preparing data for projected volume kube-api-access-zzpht for pod openshift-network-diagnostics/network-check-target-fs6d5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:32:26.143851 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:26.143750    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht podName:e27817cd-3adb-43ef-8434-7d43af58c903 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:58.143729446 +0000 UTC m=+66.381099263 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zzpht" (UniqueName: "kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht") pod "network-check-target-fs6d5" (UID: "e27817cd-3adb-43ef-8434-7d43af58c903") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 18:32:26.315297 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:26.315212    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:26.315439 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:26.315327    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:26.315439 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:26.315212    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:26.315439 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:26.315366    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:26.315618 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:26.315470    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:26.315683 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:26.315647    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:26.506994 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:26.506958    2575 generic.go:356] "Generic (PLEG): container finished" podID="2bd43ab1-614e-4ef0-b133-4d5e4f6d949b" containerID="00c445dd00de393e8195a0a50f0ed2c39df96d7264a363bbb43812c4d7f9992f" exitCode=0
Sep 09 18:32:26.507154 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:26.507026    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmjtg" event={"ID":"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b","Type":"ContainerDied","Data":"00c445dd00de393e8195a0a50f0ed2c39df96d7264a363bbb43812c4d7f9992f"}
Sep 09 18:32:27.512344 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:27.512306    2575 generic.go:356] "Generic (PLEG): container finished" podID="2bd43ab1-614e-4ef0-b133-4d5e4f6d949b" containerID="155f9df785f72bace0f4908e64a7cf41b8aa67e37cec4711f4c496b063e5dbe7" exitCode=0
Sep 09 18:32:27.512740 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:27.512380    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmjtg" event={"ID":"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b","Type":"ContainerDied","Data":"155f9df785f72bace0f4908e64a7cf41b8aa67e37cec4711f4c496b063e5dbe7"}
Sep 09 18:32:28.314351 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:28.314309    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:28.314351 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:28.314332    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:28.314606 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:28.314446    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7vhcs" podUID="3526bb20-1563-4c64-970b-e159d015bd3d"
Sep 09 18:32:28.314606 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:28.314492    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:32:28.314606 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:28.314541    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:28.314713 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:28.314648    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fs6d5" podUID="e27817cd-3adb-43ef-8434-7d43af58c903"
Sep 09 18:32:28.518683 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:28.518642    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pmjtg" event={"ID":"2bd43ab1-614e-4ef0-b133-4d5e4f6d949b","Type":"ContainerStarted","Data":"395a0c9016400a883aae41b896fd18b6d016432e97d42fb19a0f125eecf8185b"}
Sep 09 18:32:28.539470 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:28.539413    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pmjtg" podStartSLOduration=12.737764668 podStartE2EDuration="36.539399281s" totalImagesPullingTime="23.801634613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:28.539151676 +0000 UTC m=+36.776521490" watchObservedRunningTime="2026-09-09 18:32:28.539399281 +0000 UTC m=+36.776769095"
Sep 09 18:32:29.059578 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.059490    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-63.ec2.internal" event="NodeReady"
Sep 09 18:32:29.059722 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.059644    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 18:32:29.103129 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.103097    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lpmq9"]
Sep 09 18:32:29.154041 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.153993    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xsjjs"]
Sep 09 18:32:29.154041 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.154042    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:29.156939 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.156914    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 18:32:29.156939 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.156928    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 18:32:29.157098 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.157073    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dgjhb\""
Sep 09 18:32:29.172589 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.172563    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xsjjs"]
Sep 09 18:32:29.172589 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.172594    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lpmq9"]
Sep 09 18:32:29.172756 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.172673    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xsjjs"
Sep 09 18:32:29.175197 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.175172    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 18:32:29.175407 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.175389    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4r989\""
Sep 09 18:32:29.175482 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.175387    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 18:32:29.175482 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.175410    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 18:32:29.266604 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.266563    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9xh9\" (UniqueName: \"kubernetes.io/projected/d7b9ed88-9b45-457d-9e8e-9863f49d267b-kube-api-access-r9xh9\") pod \"ingress-canary-xsjjs\" (UID: \"d7b9ed88-9b45-457d-9e8e-9863f49d267b\") " pod="openshift-ingress-canary/ingress-canary-xsjjs"
Sep 09 18:32:29.266783 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.266687    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c571b1e-36e0-4647-a241-511ce5cfa737-config-volume\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:29.266783 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.266715    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:29.266783 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.266767    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wgxj\" (UniqueName: \"kubernetes.io/projected/4c571b1e-36e0-4647-a241-511ce5cfa737-kube-api-access-2wgxj\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:29.266892 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.266790    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c571b1e-36e0-4647-a241-511ce5cfa737-tmp-dir\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:29.266892 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.266813    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7b9ed88-9b45-457d-9e8e-9863f49d267b-cert\") pod \"ingress-canary-xsjjs\" (UID: \"d7b9ed88-9b45-457d-9e8e-9863f49d267b\") " pod="openshift-ingress-canary/ingress-canary-xsjjs"
Sep 09 18:32:29.368045 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.368009    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wgxj\" (UniqueName: \"kubernetes.io/projected/4c571b1e-36e0-4647-a241-511ce5cfa737-kube-api-access-2wgxj\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:29.368045 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.368048    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c571b1e-36e0-4647-a241-511ce5cfa737-tmp-dir\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:29.368306 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.368142    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7b9ed88-9b45-457d-9e8e-9863f49d267b-cert\") pod \"ingress-canary-xsjjs\" (UID: \"d7b9ed88-9b45-457d-9e8e-9863f49d267b\") " pod="openshift-ingress-canary/ingress-canary-xsjjs"
Sep 09 18:32:29.368306 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.368235    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9xh9\" (UniqueName: \"kubernetes.io/projected/d7b9ed88-9b45-457d-9e8e-9863f49d267b-kube-api-access-r9xh9\") pod \"ingress-canary-xsjjs\" (UID: \"d7b9ed88-9b45-457d-9e8e-9863f49d267b\") " pod="openshift-ingress-canary/ingress-canary-xsjjs"
Sep 09 18:32:29.368306 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.368301    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c571b1e-36e0-4647-a241-511ce5cfa737-config-volume\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:29.368448 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.368326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:29.368448 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.368390    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c571b1e-36e0-4647-a241-511ce5cfa737-tmp-dir\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:29.368579 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:29.368462    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:32:29.368579 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:29.368554    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls podName:4c571b1e-36e0-4647-a241-511ce5cfa737 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:29.868533862 +0000 UTC m=+38.105903669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls") pod "dns-default-lpmq9" (UID: "4c571b1e-36e0-4647-a241-511ce5cfa737") : secret "dns-default-metrics-tls" not found
Sep 09 18:32:29.368838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.368805    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4c571b1e-36e0-4647-a241-511ce5cfa737-config-volume\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:29.372427 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.372400    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7b9ed88-9b45-457d-9e8e-9863f49d267b-cert\") pod \"ingress-canary-xsjjs\" (UID: \"d7b9ed88-9b45-457d-9e8e-9863f49d267b\") " pod="openshift-ingress-canary/ingress-canary-xsjjs"
Sep 09 18:32:29.376447 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.376427    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wgxj\" (UniqueName: \"kubernetes.io/projected/4c571b1e-36e0-4647-a241-511ce5cfa737-kube-api-access-2wgxj\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:29.376583 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.376563    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9xh9\" (UniqueName: \"kubernetes.io/projected/d7b9ed88-9b45-457d-9e8e-9863f49d267b-kube-api-access-r9xh9\") pod \"ingress-canary-xsjjs\" (UID: \"d7b9ed88-9b45-457d-9e8e-9863f49d267b\") " pod="openshift-ingress-canary/ingress-canary-xsjjs"
Sep 09 18:32:29.480849 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.480805    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xsjjs"
Sep 09 18:32:29.794516 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.794418    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xsjjs"]
Sep 09 18:32:29.798636 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:32:29.798606    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b9ed88_9b45_457d_9e8e_9863f49d267b.slice/crio-f6cd598de51aec23de06e13b96f4406ed9162846562d1a8a77e73d4ab6b89213 WatchSource:0}: Error finding container f6cd598de51aec23de06e13b96f4406ed9162846562d1a8a77e73d4ab6b89213: Status 404 returned error can't find the container with id f6cd598de51aec23de06e13b96f4406ed9162846562d1a8a77e73d4ab6b89213
Sep 09 18:32:29.871723 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:29.871526    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:29.871880 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:29.871691    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:32:29.871880 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:29.871848    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls podName:4c571b1e-36e0-4647-a241-511ce5cfa737 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:30.871831517 +0000 UTC m=+39.109201344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls") pod "dns-default-lpmq9" (UID: "4c571b1e-36e0-4647-a241-511ce5cfa737") : secret "dns-default-metrics-tls" not found
Sep 09 18:32:30.314634 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:30.314592    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:30.314634 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:30.314629    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:30.314863 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:30.314596    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:30.317302 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:30.317280    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 18:32:30.317470 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:30.317322    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 18:32:30.318274 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:30.318256    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2784s\""
Sep 09 18:32:30.318368 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:30.318276    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 18:32:30.318466 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:30.318448    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 18:32:30.318528 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:30.318493    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vdbl9\""
Sep 09 18:32:30.525476 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:30.525434    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xsjjs" event={"ID":"d7b9ed88-9b45-457d-9e8e-9863f49d267b","Type":"ContainerStarted","Data":"f6cd598de51aec23de06e13b96f4406ed9162846562d1a8a77e73d4ab6b89213"}
Sep 09 18:32:30.880219 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:30.880174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:30.880646 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:30.880335    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:32:30.880646 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:30.880420    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls podName:4c571b1e-36e0-4647-a241-511ce5cfa737 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:32.880404329 +0000 UTC m=+41.117774138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls") pod "dns-default-lpmq9" (UID: "4c571b1e-36e0-4647-a241-511ce5cfa737") : secret "dns-default-metrics-tls" not found
Sep 09 18:32:32.893859 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:32.893813    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:32.894256 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:32.893965    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:32:32.894256 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:32.894035    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls podName:4c571b1e-36e0-4647-a241-511ce5cfa737 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:36.894019281 +0000 UTC m=+45.131389105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls") pod "dns-default-lpmq9" (UID: "4c571b1e-36e0-4647-a241-511ce5cfa737") : secret "dns-default-metrics-tls" not found
Sep 09 18:32:36.541894 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:36.541849    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xsjjs" event={"ID":"d7b9ed88-9b45-457d-9e8e-9863f49d267b","Type":"ContainerStarted","Data":"5a24dac053b554fa49d0ba587fcb439c75f62a0084cb3f87589fd83e962ed097"}
Sep 09 18:32:36.556996 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:36.556938    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xsjjs" podStartSLOduration=1.545898707 podStartE2EDuration="7.556892298s" totalImagesPullingTime="6.010993591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:32:36.555975812 +0000 UTC m=+44.793345629" watchObservedRunningTime="2026-09-09 18:32:36.556892298 +0000 UTC m=+44.794262114"
Sep 09 18:32:36.926190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:36.926126    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:36.926381 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:36.926289    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:32:36.926381 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:36.926374    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls podName:4c571b1e-36e0-4647-a241-511ce5cfa737 nodeName:}" failed. No retries permitted until 2026-09-09 18:32:44.926355575 +0000 UTC m=+53.163725402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls") pod "dns-default-lpmq9" (UID: "4c571b1e-36e0-4647-a241-511ce5cfa737") : secret "dns-default-metrics-tls" not found
Sep 09 18:32:44.984087 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:44.984040    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:32:44.984650 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:44.984233    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:32:44.984650 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:44.984330    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls podName:4c571b1e-36e0-4647-a241-511ce5cfa737 nodeName:}" failed. No retries permitted until 2026-09-09 18:33:00.984306385 +0000 UTC m=+69.221676199 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls") pod "dns-default-lpmq9" (UID: "4c571b1e-36e0-4647-a241-511ce5cfa737") : secret "dns-default-metrics-tls" not found
Sep 09 18:32:54.511138 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:54.511109    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-srx2c"
Sep 09 18:32:57.977710 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:57.977666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:57.980303 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:57.980283    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 18:32:57.991624 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:57.991596    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3526bb20-1563-4c64-970b-e159d015bd3d-original-pull-secret\") pod \"global-pull-secret-syncer-7vhcs\" (UID: \"3526bb20-1563-4c64-970b-e159d015bd3d\") " pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:58.078612 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:58.078572    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:32:58.082232 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:58.082200    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 18:32:58.088818 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:58.088791    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 18:32:58.088953 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:32:58.088869    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs podName:a83d9228-1114-4294-b196-63f6bd1a0fdd nodeName:}" failed. No retries permitted until 2026-09-09 18:34:02.0888521 +0000 UTC m=+130.326221894 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs") pod "network-metrics-daemon-kdnq9" (UID: "a83d9228-1114-4294-b196-63f6bd1a0fdd") : secret "metrics-daemon-secret" not found
Sep 09 18:32:58.179108 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:58.179060    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpht\" (UniqueName: \"kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht\") pod \"network-check-target-fs6d5\" (UID: \"e27817cd-3adb-43ef-8434-7d43af58c903\") " pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:58.182072 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:58.182046    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 18:32:58.192422 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:58.192387    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 18:32:58.202866 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:58.202831    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzpht\" (UniqueName: \"kubernetes.io/projected/e27817cd-3adb-43ef-8434-7d43af58c903-kube-api-access-zzpht\") pod \"network-check-target-fs6d5\" (UID: \"e27817cd-3adb-43ef-8434-7d43af58c903\") " pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:58.228359 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:58.228264    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7vhcs"
Sep 09 18:32:58.242042 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:58.242012    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vdbl9\""
Sep 09 18:32:58.249939 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:58.249914    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:32:58.542528 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:58.542422    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7vhcs"]
Sep 09 18:32:58.545479 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:32:58.545447    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3526bb20_1563_4c64_970b_e159d015bd3d.slice/crio-ae8ce9cd17195a3e584831ca1add71014d2ea7f05e23f205db6e55a8ff649266 WatchSource:0}: Error finding container ae8ce9cd17195a3e584831ca1add71014d2ea7f05e23f205db6e55a8ff649266: Status 404 returned error can't find the container with id ae8ce9cd17195a3e584831ca1add71014d2ea7f05e23f205db6e55a8ff649266
Sep 09 18:32:58.565166 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:58.565134    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fs6d5"]
Sep 09 18:32:58.568325 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:32:58.568299    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27817cd_3adb_43ef_8434_7d43af58c903.slice/crio-e5f201aebb79813fb4440b6028568fd193bb0dabba0389c9205fc440426a15ef WatchSource:0}: Error finding container e5f201aebb79813fb4440b6028568fd193bb0dabba0389c9205fc440426a15ef: Status 404 returned error can't find the container with id e5f201aebb79813fb4440b6028568fd193bb0dabba0389c9205fc440426a15ef
Sep 09 18:32:58.595091 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:58.595053    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fs6d5" event={"ID":"e27817cd-3adb-43ef-8434-7d43af58c903","Type":"ContainerStarted","Data":"e5f201aebb79813fb4440b6028568fd193bb0dabba0389c9205fc440426a15ef"}
Sep 09 18:32:58.596185 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:32:58.596160    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7vhcs" event={"ID":"3526bb20-1563-4c64-970b-e159d015bd3d","Type":"ContainerStarted","Data":"ae8ce9cd17195a3e584831ca1add71014d2ea7f05e23f205db6e55a8ff649266"}
Sep 09 18:33:00.998711 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:33:00.998661    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:33:00.999197 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:33:00.998831    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:33:00.999197 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:33:00.998916    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls podName:4c571b1e-36e0-4647-a241-511ce5cfa737 nodeName:}" failed. No retries permitted until 2026-09-09 18:33:32.998893403 +0000 UTC m=+101.236263200 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls") pod "dns-default-lpmq9" (UID: "4c571b1e-36e0-4647-a241-511ce5cfa737") : secret "dns-default-metrics-tls" not found
Sep 09 18:33:03.611476 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:33:03.611438    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fs6d5" event={"ID":"e27817cd-3adb-43ef-8434-7d43af58c903","Type":"ContainerStarted","Data":"ef318b654b9b24a598e784fc2c024d2ed823084ea4e572f2da3305e86cf84b3a"}
Sep 09 18:33:03.611963 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:33:03.611556    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:33:03.613026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:33:03.613004    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7vhcs" event={"ID":"3526bb20-1563-4c64-970b-e159d015bd3d","Type":"ContainerStarted","Data":"84a0caf9439f8ffedcc47ca748eff886ea2569fbc1367b2c7c78ef46fb4c7fd2"}
Sep 09 18:33:03.626643 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:33:03.626590    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fs6d5" podStartSLOduration=66.923629479 podStartE2EDuration="1m11.626574281s" totalImagesPullingTime="4.702944802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:33:03.625950623 +0000 UTC m=+71.863320439" watchObservedRunningTime="2026-09-09 18:33:03.626574281 +0000 UTC m=+71.863944089"
Sep 09 18:33:03.641867 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:33:03.641809    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7vhcs" podStartSLOduration=66.913275699 podStartE2EDuration="1m11.641791589s" totalImagesPullingTime="4.72851589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:33:03.641386641 +0000 UTC m=+71.878756456" watchObservedRunningTime="2026-09-09 18:33:03.641791589 +0000 UTC m=+71.879161403"
Sep 09 18:33:33.018981 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:33:33.018926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:33:33.019601 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:33:33.019118    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:33:33.019601 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:33:33.019220    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls podName:4c571b1e-36e0-4647-a241-511ce5cfa737 nodeName:}" failed. No retries permitted until 2026-09-09 18:34:37.019196687 +0000 UTC m=+165.256566480 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls") pod "dns-default-lpmq9" (UID: "4c571b1e-36e0-4647-a241-511ce5cfa737") : secret "dns-default-metrics-tls" not found
Sep 09 18:33:34.618424 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:33:34.618390    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fs6d5"
Sep 09 18:34:02.112571 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:34:02.112489    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:34:02.113066 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:34:02.112628    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 18:34:02.113066 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:34:02.112706    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs podName:a83d9228-1114-4294-b196-63f6bd1a0fdd nodeName:}" failed. No retries permitted until 2026-09-09 18:36:04.11268722 +0000 UTC m=+252.350057030 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs") pod "network-metrics-daemon-kdnq9" (UID: "a83d9228-1114-4294-b196-63f6bd1a0fdd") : secret "metrics-daemon-secret" not found
Sep 09 18:34:32.163626 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:34:32.163580    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-lpmq9" podUID="4c571b1e-36e0-4647-a241-511ce5cfa737"
Sep 09 18:34:32.825804 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:34:32.825773    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:34:33.334766 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:34:33.334722    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kdnq9" podUID="a83d9228-1114-4294-b196-63f6bd1a0fdd"
Sep 09 18:34:37.057462 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:34:37.057399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:34:37.057930 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:34:37.057587    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 18:34:37.057930 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:34:37.057660    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls podName:4c571b1e-36e0-4647-a241-511ce5cfa737 nodeName:}" failed. No retries permitted until 2026-09-09 18:36:39.057642152 +0000 UTC m=+287.295011944 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls") pod "dns-default-lpmq9" (UID: "4c571b1e-36e0-4647-a241-511ce5cfa737") : secret "dns-default-metrics-tls" not found
Sep 09 18:34:46.315171 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:34:46.315080    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:35:18.771405 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.771366    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"]
Sep 09 18:35:18.773575 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.773545    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-54rfc"]
Sep 09 18:35:18.773710 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.773592    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:18.775396 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.775378    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-54rfc"
Sep 09 18:35:18.778681 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.778651    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 18:35:18.778681 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.778672    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 18:35:18.778868 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.778749    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 18:35:18.778868 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.778773    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hr7ml\""
Sep 09 18:35:18.778971 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.778955    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 18:35:18.779044 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.779026    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-fk2gh\""
Sep 09 18:35:18.780196 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.780175    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-nmt4x"]
Sep 09 18:35:18.782338 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.782258    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"]
Sep 09 18:35:18.782415 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.782368    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:18.784629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.784608    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"]
Sep 09 18:35:18.784749 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.784694    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"
Sep 09 18:35:18.784749 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.784723    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 18:35:18.785050 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.785026    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 18:35:18.785132 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.785052    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-mxhg9\""
Sep 09 18:35:18.785132 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.785076    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 18:35:18.785226 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.785147    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 18:35:18.787883 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.787713    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nfhv2\""
Sep 09 18:35:18.787883 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.787805    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 09 18:35:18.787883 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.787810    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 09 18:35:18.788296 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.788270    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 09 18:35:18.789866 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.789835    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 18:35:18.790677 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.790658    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-54rfc"]
Sep 09 18:35:18.792947 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.792925    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-nmt4x"]
Sep 09 18:35:18.804079 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.804050    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"]
Sep 09 18:35:18.843900 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.843862    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jnkr9\" (UID: \"9598d5e4-eb11-4747-b7e7-c1b9681161b3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"
Sep 09 18:35:18.843900 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.843903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2848f11-65c4-4624-8cae-a9c7de831abf-serving-cert\") pod \"console-operator-85b454874b-nmt4x\" (UID: \"b2848f11-65c4-4624-8cae-a9c7de831abf\") " pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:18.844117 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.843932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:18.844117 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.843979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2848f11-65c4-4624-8cae-a9c7de831abf-config\") pod \"console-operator-85b454874b-nmt4x\" (UID: \"b2848f11-65c4-4624-8cae-a9c7de831abf\") " pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:18.844117 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.844050    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb675\" (UniqueName: \"kubernetes.io/projected/307b1285-fa18-44cd-b7a2-b2cdd8e260ea-kube-api-access-xb675\") pod \"network-check-source-685456655f-54rfc\" (UID: \"307b1285-fa18-44cd-b7a2-b2cdd8e260ea\") " pod="openshift-network-diagnostics/network-check-source-685456655f-54rfc"
Sep 09 18:35:18.844117 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.844080    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzcvg\" (UniqueName: \"kubernetes.io/projected/b2848f11-65c4-4624-8cae-a9c7de831abf-kube-api-access-xzcvg\") pod \"console-operator-85b454874b-nmt4x\" (UID: \"b2848f11-65c4-4624-8cae-a9c7de831abf\") " pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:18.844238 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.844116    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g9cg\" (UniqueName: \"kubernetes.io/projected/9598d5e4-eb11-4747-b7e7-c1b9681161b3-kube-api-access-4g9cg\") pod \"cluster-samples-operator-7cdcfcf76c-jnkr9\" (UID: \"9598d5e4-eb11-4747-b7e7-c1b9681161b3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"
Sep 09 18:35:18.844238 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.844151    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdztz\" (UniqueName: \"kubernetes.io/projected/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-kube-api-access-kdztz\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:18.844238 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.844189    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:18.844238 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.844211    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2848f11-65c4-4624-8cae-a9c7de831abf-trusted-ca\") pod \"console-operator-85b454874b-nmt4x\" (UID: \"b2848f11-65c4-4624-8cae-a9c7de831abf\") " pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:18.872675 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.872644    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9ftgh"]
Sep 09 18:35:18.874793 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.874777    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vz6kz"]
Sep 09 18:35:18.874916 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.874898    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9ftgh"
Sep 09 18:35:18.876714 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.876690    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:18.877244 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.877217    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 18:35:18.877517 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.877482    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-4zhr2\""
Sep 09 18:35:18.877671 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.877655    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 18:35:18.879032 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.879018    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 18:35:18.879334 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.879316    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 18:35:18.879415 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.879382    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 18:35:18.879736 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.879684    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 18:35:18.880583 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.880551    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mkklq\""
Sep 09 18:35:18.883893 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.883869    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 18:35:18.886338 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.886312    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9ftgh"]
Sep 09 18:35:18.887413 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.887391    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vz6kz"]
Sep 09 18:35:18.945435 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945399    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82752ac-77f8-44e5-a619-d1e1008923ce-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:18.945683 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945456    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f82752ac-77f8-44e5-a619-d1e1008923ce-snapshots\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:18.945683 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945478    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb675\" (UniqueName: \"kubernetes.io/projected/307b1285-fa18-44cd-b7a2-b2cdd8e260ea-kube-api-access-xb675\") pod \"network-check-source-685456655f-54rfc\" (UID: \"307b1285-fa18-44cd-b7a2-b2cdd8e260ea\") " pod="openshift-network-diagnostics/network-check-source-685456655f-54rfc"
Sep 09 18:35:18.945683 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzcvg\" (UniqueName: \"kubernetes.io/projected/b2848f11-65c4-4624-8cae-a9c7de831abf-kube-api-access-xzcvg\") pod \"console-operator-85b454874b-nmt4x\" (UID: \"b2848f11-65c4-4624-8cae-a9c7de831abf\") " pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:18.945683 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945542    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f82752ac-77f8-44e5-a619-d1e1008923ce-serving-cert\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:18.945683 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945559    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljpf8\" (UniqueName: \"kubernetes.io/projected/f82752ac-77f8-44e5-a619-d1e1008923ce-kube-api-access-ljpf8\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:18.945683 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945580    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g9cg\" (UniqueName: \"kubernetes.io/projected/9598d5e4-eb11-4747-b7e7-c1b9681161b3-kube-api-access-4g9cg\") pod \"cluster-samples-operator-7cdcfcf76c-jnkr9\" (UID: \"9598d5e4-eb11-4747-b7e7-c1b9681161b3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"
Sep 09 18:35:18.945683 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945598    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdztz\" (UniqueName: \"kubernetes.io/projected/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-kube-api-access-kdztz\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:18.945683 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:18.946078 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945698    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2848f11-65c4-4624-8cae-a9c7de831abf-trusted-ca\") pod \"console-operator-85b454874b-nmt4x\" (UID: \"b2848f11-65c4-4624-8cae-a9c7de831abf\") " pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:18.946078 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945737    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jnkr9\" (UID: \"9598d5e4-eb11-4747-b7e7-c1b9681161b3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"
Sep 09 18:35:18.946078 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945756    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2848f11-65c4-4624-8cae-a9c7de831abf-serving-cert\") pod \"console-operator-85b454874b-nmt4x\" (UID: \"b2848f11-65c4-4624-8cae-a9c7de831abf\") " pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:18.946078 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:18.945769    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 18:35:18.946078 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945774    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:18.946078 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:18.945831    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls podName:b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:19.445812498 +0000 UTC m=+207.683182297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-7jqpl" (UID: "b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8") : secret "cluster-monitoring-operator-tls" not found
Sep 09 18:35:18.946078 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:18.945864    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 18:35:18.946078 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:18.945944    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls podName:9598d5e4-eb11-4747-b7e7-c1b9681161b3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:19.445916896 +0000 UTC m=+207.683286694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jnkr9" (UID: "9598d5e4-eb11-4747-b7e7-c1b9681161b3") : secret "samples-operator-tls" not found
Sep 09 18:35:18.946078 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945869    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2848f11-65c4-4624-8cae-a9c7de831abf-config\") pod \"console-operator-85b454874b-nmt4x\" (UID: \"b2848f11-65c4-4624-8cae-a9c7de831abf\") " pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:18.946078 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.945988    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f82752ac-77f8-44e5-a619-d1e1008923ce-tmp\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:18.946078 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.946049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82752ac-77f8-44e5-a619-d1e1008923ce-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:18.946078 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.946086    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c747\" (UniqueName: \"kubernetes.io/projected/87708efc-02ee-4a2a-88c6-f519a08db330-kube-api-access-7c747\") pod \"volume-data-source-validator-695c895bd6-9ftgh\" (UID: \"87708efc-02ee-4a2a-88c6-f519a08db330\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9ftgh"
Sep 09 18:35:18.946699 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.946565    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2848f11-65c4-4624-8cae-a9c7de831abf-config\") pod \"console-operator-85b454874b-nmt4x\" (UID: \"b2848f11-65c4-4624-8cae-a9c7de831abf\") " pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:18.946699 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.946648    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:18.946791 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.946738    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b2848f11-65c4-4624-8cae-a9c7de831abf-trusted-ca\") pod \"console-operator-85b454874b-nmt4x\" (UID: \"b2848f11-65c4-4624-8cae-a9c7de831abf\") " pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:18.948298 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.948279    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b2848f11-65c4-4624-8cae-a9c7de831abf-serving-cert\") pod \"console-operator-85b454874b-nmt4x\" (UID: \"b2848f11-65c4-4624-8cae-a9c7de831abf\") " pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:18.954182 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.954153    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb675\" (UniqueName: \"kubernetes.io/projected/307b1285-fa18-44cd-b7a2-b2cdd8e260ea-kube-api-access-xb675\") pod \"network-check-source-685456655f-54rfc\" (UID: \"307b1285-fa18-44cd-b7a2-b2cdd8e260ea\") " pod="openshift-network-diagnostics/network-check-source-685456655f-54rfc"
Sep 09 18:35:18.954347 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.954260    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g9cg\" (UniqueName: \"kubernetes.io/projected/9598d5e4-eb11-4747-b7e7-c1b9681161b3-kube-api-access-4g9cg\") pod \"cluster-samples-operator-7cdcfcf76c-jnkr9\" (UID: \"9598d5e4-eb11-4747-b7e7-c1b9681161b3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"
Sep 09 18:35:18.954445 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.954425    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzcvg\" (UniqueName: \"kubernetes.io/projected/b2848f11-65c4-4624-8cae-a9c7de831abf-kube-api-access-xzcvg\") pod \"console-operator-85b454874b-nmt4x\" (UID: \"b2848f11-65c4-4624-8cae-a9c7de831abf\") " pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:18.954522 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:18.954448    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdztz\" (UniqueName: \"kubernetes.io/projected/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-kube-api-access-kdztz\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:19.046671 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.046561    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f82752ac-77f8-44e5-a619-d1e1008923ce-snapshots\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:19.046671 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.046609    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f82752ac-77f8-44e5-a619-d1e1008923ce-serving-cert\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:19.046671 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.046632    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljpf8\" (UniqueName: \"kubernetes.io/projected/f82752ac-77f8-44e5-a619-d1e1008923ce-kube-api-access-ljpf8\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:19.046959 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.046734    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f82752ac-77f8-44e5-a619-d1e1008923ce-tmp\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:19.046959 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.046760    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82752ac-77f8-44e5-a619-d1e1008923ce-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:19.046959 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.046786    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c747\" (UniqueName: \"kubernetes.io/projected/87708efc-02ee-4a2a-88c6-f519a08db330-kube-api-access-7c747\") pod \"volume-data-source-validator-695c895bd6-9ftgh\" (UID: \"87708efc-02ee-4a2a-88c6-f519a08db330\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9ftgh"
Sep 09 18:35:19.046959 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.046817    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82752ac-77f8-44e5-a619-d1e1008923ce-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:19.047318 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.047290    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f82752ac-77f8-44e5-a619-d1e1008923ce-tmp\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:19.047407 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.047333    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f82752ac-77f8-44e5-a619-d1e1008923ce-snapshots\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:19.047466 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.047417    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82752ac-77f8-44e5-a619-d1e1008923ce-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:19.047621 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.047602    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82752ac-77f8-44e5-a619-d1e1008923ce-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:19.049220 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.049198    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f82752ac-77f8-44e5-a619-d1e1008923ce-serving-cert\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:19.054292 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.054265    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljpf8\" (UniqueName: \"kubernetes.io/projected/f82752ac-77f8-44e5-a619-d1e1008923ce-kube-api-access-ljpf8\") pod \"insights-operator-76bcd6ccf5-vz6kz\" (UID: \"f82752ac-77f8-44e5-a619-d1e1008923ce\") " pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:19.054929 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.054908    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c747\" (UniqueName: \"kubernetes.io/projected/87708efc-02ee-4a2a-88c6-f519a08db330-kube-api-access-7c747\") pod \"volume-data-source-validator-695c895bd6-9ftgh\" (UID: \"87708efc-02ee-4a2a-88c6-f519a08db330\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9ftgh"
Sep 09 18:35:19.093313 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.093272    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-54rfc"
Sep 09 18:35:19.100240 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.100209    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:19.185830 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.185802    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9ftgh"
Sep 09 18:35:19.193238 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.193215    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz"
Sep 09 18:35:19.365746 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.365711    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-54rfc"]
Sep 09 18:35:19.369278 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:35:19.369243    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307b1285_fa18_44cd_b7a2_b2cdd8e260ea.slice/crio-a0d1c005714123b1cb4b9dee8bd86e11ed115893bd7d1fc9765b5b1137379f22 WatchSource:0}: Error finding container a0d1c005714123b1cb4b9dee8bd86e11ed115893bd7d1fc9765b5b1137379f22: Status 404 returned error can't find the container with id a0d1c005714123b1cb4b9dee8bd86e11ed115893bd7d1fc9765b5b1137379f22
Sep 09 18:35:19.385698 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.385636    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-nmt4x"]
Sep 09 18:35:19.402664 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:35:19.402632    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2848f11_65c4_4624_8cae_a9c7de831abf.slice/crio-cababec2fa1174249db053d8d913e5c4827ba6702a2f0be6ac45e9e5bf7de6ee WatchSource:0}: Error finding container cababec2fa1174249db053d8d913e5c4827ba6702a2f0be6ac45e9e5bf7de6ee: Status 404 returned error can't find the container with id cababec2fa1174249db053d8d913e5c4827ba6702a2f0be6ac45e9e5bf7de6ee
Sep 09 18:35:19.451284 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.451253    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jnkr9\" (UID: \"9598d5e4-eb11-4747-b7e7-c1b9681161b3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"
Sep 09 18:35:19.451439 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.451359    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:19.451439 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:19.451412    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 18:35:19.451576 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:19.451445    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 18:35:19.451576 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:19.451477    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls podName:9598d5e4-eb11-4747-b7e7-c1b9681161b3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:20.451459162 +0000 UTC m=+208.688828972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jnkr9" (UID: "9598d5e4-eb11-4747-b7e7-c1b9681161b3") : secret "samples-operator-tls" not found
Sep 09 18:35:19.451576 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:19.451522    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls podName:b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:20.45148541 +0000 UTC m=+208.688855203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-7jqpl" (UID: "b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8") : secret "cluster-monitoring-operator-tls" not found
Sep 09 18:35:19.460165 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.460134    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9ftgh"]
Sep 09 18:35:19.464463 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:35:19.464430    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87708efc_02ee_4a2a_88c6_f519a08db330.slice/crio-1178ea01d4c037d1ef3497d7ac9bfa65b2da73c93b638902efedc714518fd985 WatchSource:0}: Error finding container 1178ea01d4c037d1ef3497d7ac9bfa65b2da73c93b638902efedc714518fd985: Status 404 returned error can't find the container with id 1178ea01d4c037d1ef3497d7ac9bfa65b2da73c93b638902efedc714518fd985
Sep 09 18:35:19.472763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.472732    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-vz6kz"]
Sep 09 18:35:19.476015 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:35:19.475986    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82752ac_77f8_44e5_a619_d1e1008923ce.slice/crio-a889f058329e3f9855b67862c761bf2a49e05d6f77c4c26575531342351dbe4b WatchSource:0}: Error finding container a889f058329e3f9855b67862c761bf2a49e05d6f77c4c26575531342351dbe4b: Status 404 returned error can't find the container with id a889f058329e3f9855b67862c761bf2a49e05d6f77c4c26575531342351dbe4b
Sep 09 18:35:19.939985 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.939944    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz" event={"ID":"f82752ac-77f8-44e5-a619-d1e1008923ce","Type":"ContainerStarted","Data":"a889f058329e3f9855b67862c761bf2a49e05d6f77c4c26575531342351dbe4b"}
Sep 09 18:35:19.941133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.941097    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9ftgh" event={"ID":"87708efc-02ee-4a2a-88c6-f519a08db330","Type":"ContainerStarted","Data":"1178ea01d4c037d1ef3497d7ac9bfa65b2da73c93b638902efedc714518fd985"}
Sep 09 18:35:19.942614 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.942586    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-54rfc" event={"ID":"307b1285-fa18-44cd-b7a2-b2cdd8e260ea","Type":"ContainerStarted","Data":"05f3159b33cac8b1f1ca6cbd4a4860a6f735296edce3e0ce48784c67d5031c7b"}
Sep 09 18:35:19.942713 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.942633    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-54rfc" event={"ID":"307b1285-fa18-44cd-b7a2-b2cdd8e260ea","Type":"ContainerStarted","Data":"a0d1c005714123b1cb4b9dee8bd86e11ed115893bd7d1fc9765b5b1137379f22"}
Sep 09 18:35:19.943719 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.943694    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-nmt4x" event={"ID":"b2848f11-65c4-4624-8cae-a9c7de831abf","Type":"ContainerStarted","Data":"cababec2fa1174249db053d8d913e5c4827ba6702a2f0be6ac45e9e5bf7de6ee"}
Sep 09 18:35:19.956877 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:19.956819    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-54rfc" podStartSLOduration=1.956802139 podStartE2EDuration="1.956802139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:35:19.956334186 +0000 UTC m=+208.193704002" watchObservedRunningTime="2026-09-09 18:35:19.956802139 +0000 UTC m=+208.194171953"
Sep 09 18:35:20.460521 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:20.459880    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jnkr9\" (UID: \"9598d5e4-eb11-4747-b7e7-c1b9681161b3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"
Sep 09 18:35:20.460521 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:20.460013    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:20.460521 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:20.460142    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 18:35:20.460521 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:20.460208    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls podName:b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:22.460187833 +0000 UTC m=+210.697557632 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-7jqpl" (UID: "b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8") : secret "cluster-monitoring-operator-tls" not found
Sep 09 18:35:20.460857 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:20.460562    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 18:35:20.460857 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:20.460625    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls podName:9598d5e4-eb11-4747-b7e7-c1b9681161b3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:22.460606999 +0000 UTC m=+210.697976797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jnkr9" (UID: "9598d5e4-eb11-4747-b7e7-c1b9681161b3") : secret "samples-operator-tls" not found
Sep 09 18:35:21.953825 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:21.953782    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9ftgh" event={"ID":"87708efc-02ee-4a2a-88c6-f519a08db330","Type":"ContainerStarted","Data":"b7b773da90c2b3313a22b756d57e5563eedd35b31416299ecbb7d74fcf51e739"}
Sep 09 18:35:22.477727 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.477686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:22.477946 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.477740    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jnkr9\" (UID: \"9598d5e4-eb11-4747-b7e7-c1b9681161b3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"
Sep 09 18:35:22.477946 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:22.477880    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 18:35:22.477946 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:22.477881    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 18:35:22.477946 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:22.477936    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls podName:9598d5e4-eb11-4747-b7e7-c1b9681161b3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:26.477922063 +0000 UTC m=+214.715291856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jnkr9" (UID: "9598d5e4-eb11-4747-b7e7-c1b9681161b3") : secret "samples-operator-tls" not found
Sep 09 18:35:22.477946 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:22.477949    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls podName:b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:26.477943644 +0000 UTC m=+214.715313436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-7jqpl" (UID: "b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8") : secret "cluster-monitoring-operator-tls" not found
Sep 09 18:35:22.859654 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.859591    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-9ftgh" podStartSLOduration=3.316498399 podStartE2EDuration="4.859570693s" totalImagesPullingTime="1.543072294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:35:21.967968897 +0000 UTC m=+210.205338712" watchObservedRunningTime="2026-09-09 18:35:22.859570693 +0000 UTC m=+211.096940610"
Sep 09 18:35:22.859980 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.859964    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tlfq9"]
Sep 09 18:35:22.862414 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.862383    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:22.864772 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.864746    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 18:35:22.864919 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.864801    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 18:35:22.866088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.866054    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nn2fg\""
Sep 09 18:35:22.874374 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.874347    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tlfq9"]
Sep 09 18:35:22.959586 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.959557    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/0.log"
Sep 09 18:35:22.960037 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.959599    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2848f11-65c4-4624-8cae-a9c7de831abf" containerID="ee5a0c02f335fa5ebbf65ad4663e12128da048ec230ea351db2934f37f07e133" exitCode=255
Sep 09 18:35:22.960037 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.959704    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-nmt4x" event={"ID":"b2848f11-65c4-4624-8cae-a9c7de831abf","Type":"ContainerDied","Data":"ee5a0c02f335fa5ebbf65ad4663e12128da048ec230ea351db2934f37f07e133"}
Sep 09 18:35:22.960037 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.959905    2575 scope.go:117] "RemoveContainer" containerID="ee5a0c02f335fa5ebbf65ad4663e12128da048ec230ea351db2934f37f07e133"
Sep 09 18:35:22.961359 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.961330    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz" event={"ID":"f82752ac-77f8-44e5-a619-d1e1008923ce","Type":"ContainerStarted","Data":"e3576ee5ac68e517ce865114bc0e7d55e3a67a9e974551b8b421794175f35aa0"}
Sep 09 18:35:22.981398 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.981363    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdjf6\" (UniqueName: \"kubernetes.io/projected/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-api-access-bdjf6\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:22.981575 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.981419    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-crio-socket\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:22.981575 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.981516    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-data-volume\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:22.981575 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.981559    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:22.981685 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.981614    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:22.988464 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:22.988408    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz" podStartSLOduration=2.352211076 podStartE2EDuration="4.988390141s" totalImagesPullingTime="2.636179065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:35:22.987611474 +0000 UTC m=+211.224981288" watchObservedRunningTime="2026-09-09 18:35:22.988390141 +0000 UTC m=+211.225759956"
Sep 09 18:35:23.082399 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.082362    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-crio-socket\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:23.082567 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.082449    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-data-volume\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:23.082567 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.082465    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:23.082567 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.082519    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-crio-socket\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:23.082751 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.082574    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:23.082751 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:23.082620    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 18:35:23.082751 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:23.082710    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm podName:dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:23.582689104 +0000 UTC m=+211.820058910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tlfq9" (UID: "dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3") : configmap "kube-rbac-proxy" not found
Sep 09 18:35:23.082900 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.082854    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-data-volume\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:23.082958 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.082623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdjf6\" (UniqueName: \"kubernetes.io/projected/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-api-access-bdjf6\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:23.083011 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:23.082966    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:23.083062 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:23.083013    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls podName:dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:23.582996825 +0000 UTC m=+211.820366626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tlfq9" (UID: "dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:23.092516 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.092466    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdjf6\" (UniqueName: \"kubernetes.io/projected/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-api-access-bdjf6\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:23.587278 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.587217    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:23.587469 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.587315    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:23.587469 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:23.587343    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 18:35:23.587469 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:23.587410    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm podName:dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:24.587391293 +0000 UTC m=+212.824761098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tlfq9" (UID: "dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3") : configmap "kube-rbac-proxy" not found
Sep 09 18:35:23.587469 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:23.587459    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:23.587650 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:23.587536    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls podName:dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:24.587521578 +0000 UTC m=+212.824891375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tlfq9" (UID: "dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:23.966317 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.966285    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/1.log"
Sep 09 18:35:23.966746 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.966715    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/0.log"
Sep 09 18:35:23.966795 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.966750    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2848f11-65c4-4624-8cae-a9c7de831abf" containerID="6a514be1231c6ff586c18d6bd27dfd425e48661507a4444caba069e9b78ac80b" exitCode=255
Sep 09 18:35:23.966835 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.966818    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-nmt4x" event={"ID":"b2848f11-65c4-4624-8cae-a9c7de831abf","Type":"ContainerDied","Data":"6a514be1231c6ff586c18d6bd27dfd425e48661507a4444caba069e9b78ac80b"}
Sep 09 18:35:23.966912 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.966901    2575 scope.go:117] "RemoveContainer" containerID="ee5a0c02f335fa5ebbf65ad4663e12128da048ec230ea351db2934f37f07e133"
Sep 09 18:35:23.967127 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:23.967105    2575 scope.go:117] "RemoveContainer" containerID="6a514be1231c6ff586c18d6bd27dfd425e48661507a4444caba069e9b78ac80b"
Sep 09 18:35:23.967307 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:23.967282    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-nmt4x_openshift-console-operator(b2848f11-65c4-4624-8cae-a9c7de831abf)\"" pod="openshift-console-operator/console-operator-85b454874b-nmt4x" podUID="b2848f11-65c4-4624-8cae-a9c7de831abf"
Sep 09 18:35:24.596860 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:24.596796    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:24.597092 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:24.596897    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:24.597092 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:24.596925    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 18:35:24.597092 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:24.596999    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm podName:dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:26.596982409 +0000 UTC m=+214.834352206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tlfq9" (UID: "dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3") : configmap "kube-rbac-proxy" not found
Sep 09 18:35:24.597092 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:24.597013    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:24.597092 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:24.597066    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls podName:dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:26.597050597 +0000 UTC m=+214.834420408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tlfq9" (UID: "dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:24.821390 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:24.821361    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lpvdl_d4099f20-afa3-482b-a22e-c4d59554f7fd/dns-node-resolver/0.log"
Sep 09 18:35:24.971924 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:24.971895    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/1.log"
Sep 09 18:35:24.972365 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:24.972219    2575 scope.go:117] "RemoveContainer" containerID="6a514be1231c6ff586c18d6bd27dfd425e48661507a4444caba069e9b78ac80b"
Sep 09 18:35:24.972421 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:24.972365    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-nmt4x_openshift-console-operator(b2848f11-65c4-4624-8cae-a9c7de831abf)\"" pod="openshift-console-operator/console-operator-85b454874b-nmt4x" podUID="b2848f11-65c4-4624-8cae-a9c7de831abf"
Sep 09 18:35:26.021620 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.021594    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t75rf_1753a297-ac2b-49cd-b9c9-459465d3d912/node-ca/0.log"
Sep 09 18:35:26.050478 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.050436    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-ksstr"]
Sep 09 18:35:26.053380 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.053353    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-ksstr"
Sep 09 18:35:26.055982 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.055956    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-crlrq\""
Sep 09 18:35:26.056139 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.056033    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 09 18:35:26.056904 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.056882    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 09 18:35:26.056993 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.056908    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 09 18:35:26.056993 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.056887    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 09 18:35:26.062137 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.062114    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-ksstr"]
Sep 09 18:35:26.108319 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.108277    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f47fd2ab-bd00-46fc-a966-7410593fe577-signing-key\") pod \"service-ca-75d99c484-ksstr\" (UID: \"f47fd2ab-bd00-46fc-a966-7410593fe577\") " pod="openshift-service-ca/service-ca-75d99c484-ksstr"
Sep 09 18:35:26.108319 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.108316    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f47fd2ab-bd00-46fc-a966-7410593fe577-signing-cabundle\") pod \"service-ca-75d99c484-ksstr\" (UID: \"f47fd2ab-bd00-46fc-a966-7410593fe577\") " pod="openshift-service-ca/service-ca-75d99c484-ksstr"
Sep 09 18:35:26.108603 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.108427    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vk5s\" (UniqueName: \"kubernetes.io/projected/f47fd2ab-bd00-46fc-a966-7410593fe577-kube-api-access-8vk5s\") pod \"service-ca-75d99c484-ksstr\" (UID: \"f47fd2ab-bd00-46fc-a966-7410593fe577\") " pod="openshift-service-ca/service-ca-75d99c484-ksstr"
Sep 09 18:35:26.209724 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.209684    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f47fd2ab-bd00-46fc-a966-7410593fe577-signing-key\") pod \"service-ca-75d99c484-ksstr\" (UID: \"f47fd2ab-bd00-46fc-a966-7410593fe577\") " pod="openshift-service-ca/service-ca-75d99c484-ksstr"
Sep 09 18:35:26.209724 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.209722    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f47fd2ab-bd00-46fc-a966-7410593fe577-signing-cabundle\") pod \"service-ca-75d99c484-ksstr\" (UID: \"f47fd2ab-bd00-46fc-a966-7410593fe577\") " pod="openshift-service-ca/service-ca-75d99c484-ksstr"
Sep 09 18:35:26.209973 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.209861    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vk5s\" (UniqueName: \"kubernetes.io/projected/f47fd2ab-bd00-46fc-a966-7410593fe577-kube-api-access-8vk5s\") pod \"service-ca-75d99c484-ksstr\" (UID: \"f47fd2ab-bd00-46fc-a966-7410593fe577\") " pod="openshift-service-ca/service-ca-75d99c484-ksstr"
Sep 09 18:35:26.210357 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.210338    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f47fd2ab-bd00-46fc-a966-7410593fe577-signing-cabundle\") pod \"service-ca-75d99c484-ksstr\" (UID: \"f47fd2ab-bd00-46fc-a966-7410593fe577\") " pod="openshift-service-ca/service-ca-75d99c484-ksstr"
Sep 09 18:35:26.212250 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.212226    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f47fd2ab-bd00-46fc-a966-7410593fe577-signing-key\") pod \"service-ca-75d99c484-ksstr\" (UID: \"f47fd2ab-bd00-46fc-a966-7410593fe577\") " pod="openshift-service-ca/service-ca-75d99c484-ksstr"
Sep 09 18:35:26.218107 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.218082    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vk5s\" (UniqueName: \"kubernetes.io/projected/f47fd2ab-bd00-46fc-a966-7410593fe577-kube-api-access-8vk5s\") pod \"service-ca-75d99c484-ksstr\" (UID: \"f47fd2ab-bd00-46fc-a966-7410593fe577\") " pod="openshift-service-ca/service-ca-75d99c484-ksstr"
Sep 09 18:35:26.363905 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.363871    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-ksstr"
Sep 09 18:35:26.512344 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.512311    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:26.512485 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.512386    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jnkr9\" (UID: \"9598d5e4-eb11-4747-b7e7-c1b9681161b3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"
Sep 09 18:35:26.512485 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:26.512419    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 18:35:26.512485 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:26.512483    2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 09 18:35:26.512617 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:26.512490    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls podName:b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:34.51247259 +0000 UTC m=+222.749842382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-7jqpl" (UID: "b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8") : secret "cluster-monitoring-operator-tls" not found
Sep 09 18:35:26.512617 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:26.512535    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls podName:9598d5e4-eb11-4747-b7e7-c1b9681161b3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:34.512523594 +0000 UTC m=+222.749893388 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-jnkr9" (UID: "9598d5e4-eb11-4747-b7e7-c1b9681161b3") : secret "samples-operator-tls" not found
Sep 09 18:35:26.613426 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.613389    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:26.613426 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.613432    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:26.613655 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:26.613540    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 18:35:26.613655 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:26.613610    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm podName:dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:30.613593001 +0000 UTC m=+218.850962794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tlfq9" (UID: "dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3") : configmap "kube-rbac-proxy" not found
Sep 09 18:35:26.613727 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:26.613661    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:26.613727 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:26.613717    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls podName:dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:30.613702106 +0000 UTC m=+218.851071898 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tlfq9" (UID: "dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:26.636807 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.636719    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-ksstr"]
Sep 09 18:35:26.640929 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:35:26.640898    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47fd2ab_bd00_46fc_a966_7410593fe577.slice/crio-b3ca60c0ab85067f301025d16a087aede8a1bb3e7179cdd847184b90c0c99cc5 WatchSource:0}: Error finding container b3ca60c0ab85067f301025d16a087aede8a1bb3e7179cdd847184b90c0c99cc5: Status 404 returned error can't find the container with id b3ca60c0ab85067f301025d16a087aede8a1bb3e7179cdd847184b90c0c99cc5
Sep 09 18:35:26.821353 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.821326    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xsjjs_d7b9ed88-9b45-457d-9e8e-9863f49d267b/serve-healthcheck-canary/0.log"
Sep 09 18:35:26.981046 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:26.980962    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-ksstr" event={"ID":"f47fd2ab-bd00-46fc-a966-7410593fe577","Type":"ContainerStarted","Data":"b3ca60c0ab85067f301025d16a087aede8a1bb3e7179cdd847184b90c0c99cc5"}
Sep 09 18:35:28.990302 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:28.990258    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-ksstr" event={"ID":"f47fd2ab-bd00-46fc-a966-7410593fe577","Type":"ContainerStarted","Data":"0e226a95c702a076953154f8ea0d491213508c4cfe81943ea66e52013e02d7fb"}
Sep 09 18:35:29.006324 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:29.006263    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-ksstr" podStartSLOduration=1.130690366 podStartE2EDuration="3.006246015s" totalImagesPullingTime="1.875555649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:35:29.005052617 +0000 UTC m=+217.242422434" watchObservedRunningTime="2026-09-09 18:35:29.006246015 +0000 UTC m=+217.243615829"
Sep 09 18:35:29.100641 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:29.100595    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:29.100793 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:29.100670    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:29.101123 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:29.101108    2575 scope.go:117] "RemoveContainer" containerID="6a514be1231c6ff586c18d6bd27dfd425e48661507a4444caba069e9b78ac80b"
Sep 09 18:35:29.101331 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:29.101308    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-nmt4x_openshift-console-operator(b2848f11-65c4-4624-8cae-a9c7de831abf)\"" pod="openshift-console-operator/console-operator-85b454874b-nmt4x" podUID="b2848f11-65c4-4624-8cae-a9c7de831abf"
Sep 09 18:35:30.652134 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:30.652082    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:30.652134 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:30.652137    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:30.652739 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:30.652228    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 18:35:30.652739 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:30.652235    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:30.652739 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:30.652318    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm podName:dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:38.652289453 +0000 UTC m=+226.889659258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tlfq9" (UID: "dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3") : configmap "kube-rbac-proxy" not found
Sep 09 18:35:30.652739 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:30.652357    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls podName:dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:38.652334452 +0000 UTC m=+226.889704252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tlfq9" (UID: "dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:34.585703 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:34.585661    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jnkr9\" (UID: \"9598d5e4-eb11-4747-b7e7-c1b9681161b3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"
Sep 09 18:35:34.586120 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:34.585742    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:34.586120 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:34.585905    2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 18:35:34.586120 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:34.585979    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls podName:b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:50.585962685 +0000 UTC m=+238.823332478 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-7jqpl" (UID: "b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8") : secret "cluster-monitoring-operator-tls" not found
Sep 09 18:35:34.588239 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:34.588207    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9598d5e4-eb11-4747-b7e7-c1b9681161b3-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-jnkr9\" (UID: \"9598d5e4-eb11-4747-b7e7-c1b9681161b3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"
Sep 09 18:35:34.706460 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:34.706414    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"
Sep 09 18:35:34.979539 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:34.979485    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9"]
Sep 09 18:35:36.017480 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:36.017440    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9" event={"ID":"9598d5e4-eb11-4747-b7e7-c1b9681161b3","Type":"ContainerStarted","Data":"c6d45b7c5e20a4dd5dd8c722822f55fdc00cd9e6ca7ec7be0219104475cfe7e4"}
Sep 09 18:35:38.025709 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:38.025673    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9" event={"ID":"9598d5e4-eb11-4747-b7e7-c1b9681161b3","Type":"ContainerStarted","Data":"6c7bff00037727510f1726f340db20ea63433a2a23099c10940100e85b933047"}
Sep 09 18:35:38.026182 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:38.025724    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9" event={"ID":"9598d5e4-eb11-4747-b7e7-c1b9681161b3","Type":"ContainerStarted","Data":"0bbd32bfcdc5d0edd98aded2a3026b46f10558fddd362dd345f1595b337262a2"}
Sep 09 18:35:38.041770 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:38.041715    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-jnkr9" podStartSLOduration=17.988208275 podStartE2EDuration="20.041698023s" totalImagesPullingTime="2.053489748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:35:38.040202448 +0000 UTC m=+226.277572263" watchObservedRunningTime="2026-09-09 18:35:38.041698023 +0000 UTC m=+226.279067838"
Sep 09 18:35:38.721315 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:38.721270    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:38.721525 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:38.721443    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:38.721525 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:38.721475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:38.721626 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:38.721537    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls podName:dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:54.721515848 +0000 UTC m=+242.958885660 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tlfq9" (UID: "dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3") : secret "insights-runtime-extractor-tls" not found
Sep 09 18:35:38.721994 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:38.721976    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:43.315439 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:43.315404    2575 scope.go:117] "RemoveContainer" containerID="6a514be1231c6ff586c18d6bd27dfd425e48661507a4444caba069e9b78ac80b"
Sep 09 18:35:44.045617 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:44.045589    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 18:35:44.046030 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:44.046015    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/1.log"
Sep 09 18:35:44.046075 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:44.046047    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2848f11-65c4-4624-8cae-a9c7de831abf" containerID="53d02c2203463f148b952b25d57ab7583d810b1aef4ceaf369e0a5fee7b1344e" exitCode=255
Sep 09 18:35:44.046109 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:44.046087    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-nmt4x" event={"ID":"b2848f11-65c4-4624-8cae-a9c7de831abf","Type":"ContainerDied","Data":"53d02c2203463f148b952b25d57ab7583d810b1aef4ceaf369e0a5fee7b1344e"}
Sep 09 18:35:44.046142 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:44.046131    2575 scope.go:117] "RemoveContainer" containerID="6a514be1231c6ff586c18d6bd27dfd425e48661507a4444caba069e9b78ac80b"
Sep 09 18:35:44.046425 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:44.046408    2575 scope.go:117] "RemoveContainer" containerID="53d02c2203463f148b952b25d57ab7583d810b1aef4ceaf369e0a5fee7b1344e"
Sep 09 18:35:44.046640 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:44.046620    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-nmt4x_openshift-console-operator(b2848f11-65c4-4624-8cae-a9c7de831abf)\"" pod="openshift-console-operator/console-operator-85b454874b-nmt4x" podUID="b2848f11-65c4-4624-8cae-a9c7de831abf"
Sep 09 18:35:45.050844 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:45.050812    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 18:35:49.100842 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:49.100797    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:49.101244 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:49.100862    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:35:49.101244 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:49.101164    2575 scope.go:117] "RemoveContainer" containerID="53d02c2203463f148b952b25d57ab7583d810b1aef4ceaf369e0a5fee7b1344e"
Sep 09 18:35:49.101332 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:49.101316    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-nmt4x_openshift-console-operator(b2848f11-65c4-4624-8cae-a9c7de831abf)\"" pod="openshift-console-operator/console-operator-85b454874b-nmt4x" podUID="b2848f11-65c4-4624-8cae-a9c7de831abf"
Sep 09 18:35:50.263289 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.263252    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"]
Sep 09 18:35:50.266409 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.266391    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.268662 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.268636    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-74w6f\""
Sep 09 18:35:50.269640 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.269616    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 18:35:50.269781 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.269698    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 18:35:50.269857 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.269801    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 18:35:50.273215 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.273197    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 18:35:50.276382 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.276359    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"]
Sep 09 18:35:50.415038 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.414999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b27f60e-eb9e-41f5-8141-6c2ed0448395-registry-tls\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.415236 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.415063    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7b27f60e-eb9e-41f5-8141-6c2ed0448395-installation-pull-secrets\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.415236 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.415119    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7b27f60e-eb9e-41f5-8141-6c2ed0448395-ca-trust-extracted\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.415236 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.415150    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b27f60e-eb9e-41f5-8141-6c2ed0448395-bound-sa-token\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.415236 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.415183    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b27f60e-eb9e-41f5-8141-6c2ed0448395-trusted-ca\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.415236 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.415211    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7b27f60e-eb9e-41f5-8141-6c2ed0448395-registry-certificates\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.415399 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.415265    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7b27f60e-eb9e-41f5-8141-6c2ed0448395-image-registry-private-configuration\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.415399 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.415304    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfvlj\" (UniqueName: \"kubernetes.io/projected/7b27f60e-eb9e-41f5-8141-6c2ed0448395-kube-api-access-sfvlj\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.516016 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.515914    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b27f60e-eb9e-41f5-8141-6c2ed0448395-registry-tls\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.516016 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.515984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7b27f60e-eb9e-41f5-8141-6c2ed0448395-installation-pull-secrets\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.516016 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.516019    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7b27f60e-eb9e-41f5-8141-6c2ed0448395-ca-trust-extracted\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.516379 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.516045    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b27f60e-eb9e-41f5-8141-6c2ed0448395-bound-sa-token\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.516379 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.516080    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b27f60e-eb9e-41f5-8141-6c2ed0448395-trusted-ca\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.516379 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.516110    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7b27f60e-eb9e-41f5-8141-6c2ed0448395-registry-certificates\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.516379 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.516162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7b27f60e-eb9e-41f5-8141-6c2ed0448395-image-registry-private-configuration\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.516379 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.516201    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfvlj\" (UniqueName: \"kubernetes.io/projected/7b27f60e-eb9e-41f5-8141-6c2ed0448395-kube-api-access-sfvlj\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.516674 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.516577    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7b27f60e-eb9e-41f5-8141-6c2ed0448395-ca-trust-extracted\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.517263 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.517232    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b27f60e-eb9e-41f5-8141-6c2ed0448395-trusted-ca\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.517263 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.517254    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7b27f60e-eb9e-41f5-8141-6c2ed0448395-registry-certificates\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.518810 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.518779    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7b27f60e-eb9e-41f5-8141-6c2ed0448395-installation-pull-secrets\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.519002 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.518981    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7b27f60e-eb9e-41f5-8141-6c2ed0448395-image-registry-private-configuration\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.519160 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.519140    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7b27f60e-eb9e-41f5-8141-6c2ed0448395-registry-tls\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.523344 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.523316    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfvlj\" (UniqueName: \"kubernetes.io/projected/7b27f60e-eb9e-41f5-8141-6c2ed0448395-kube-api-access-sfvlj\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.523435 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.523358    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b27f60e-eb9e-41f5-8141-6c2ed0448395-bound-sa-token\") pod \"image-registry-6b7bd5bbf4-6sctb\" (UID: \"7b27f60e-eb9e-41f5-8141-6c2ed0448395\") " pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.576207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.576172    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:50.617265 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.617228    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:50.619670 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.619637    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-7jqpl\" (UID: \"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:50.849886 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.849803    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"]
Sep 09 18:35:50.852829 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:35:50.852797    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b27f60e_eb9e_41f5_8141_6c2ed0448395.slice/crio-0774e707145f6920a12620d0c6f98f56e0fdd50764187297b48a6b591b2d1a10 WatchSource:0}: Error finding container 0774e707145f6920a12620d0c6f98f56e0fdd50764187297b48a6b591b2d1a10: Status 404 returned error can't find the container with id 0774e707145f6920a12620d0c6f98f56e0fdd50764187297b48a6b591b2d1a10
Sep 09 18:35:50.886381 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:50.886352    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"
Sep 09 18:35:51.071775 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:51.071733    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb" event={"ID":"7b27f60e-eb9e-41f5-8141-6c2ed0448395","Type":"ContainerStarted","Data":"797719c9a4f462afa16e0427e5c903deacc03246b04c9abbe30dfda70186bd6e"}
Sep 09 18:35:51.071989 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:51.071792    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb" event={"ID":"7b27f60e-eb9e-41f5-8141-6c2ed0448395","Type":"ContainerStarted","Data":"0774e707145f6920a12620d0c6f98f56e0fdd50764187297b48a6b591b2d1a10"}
Sep 09 18:35:51.071989 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:51.071875    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:35:51.089768 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:51.089713    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb" podStartSLOduration=1.089689581 podStartE2EDuration="1.089689581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:35:51.08839993 +0000 UTC m=+239.325769760" watchObservedRunningTime="2026-09-09 18:35:51.089689581 +0000 UTC m=+239.327059400"
Sep 09 18:35:51.162255 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:51.162217    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl"]
Sep 09 18:35:51.165145 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:35:51.165117    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9722a7d_f4e8_49ba_aa95_a0adaad2f4a8.slice/crio-dca996c28ab1e0b7e5bb6f39e0ccd5aa6a455fae00a4d05fa28a944837fddc72 WatchSource:0}: Error finding container dca996c28ab1e0b7e5bb6f39e0ccd5aa6a455fae00a4d05fa28a944837fddc72: Status 404 returned error can't find the container with id dca996c28ab1e0b7e5bb6f39e0ccd5aa6a455fae00a4d05fa28a944837fddc72
Sep 09 18:35:52.075855 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:52.075812    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl" event={"ID":"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8","Type":"ContainerStarted","Data":"dca996c28ab1e0b7e5bb6f39e0ccd5aa6a455fae00a4d05fa28a944837fddc72"}
Sep 09 18:35:53.703346 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:53.703307    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dmhfm"]
Sep 09 18:35:53.705558 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:53.705536    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dmhfm"
Sep 09 18:35:53.707978 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:53.707948    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 18:35:53.708154 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:53.708033    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xbpsl\""
Sep 09 18:35:53.713114 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:53.713083    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dmhfm"]
Sep 09 18:35:53.843884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:53.843833    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e88beb3-f59b-4503-8215-a3e8f4cde9c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dmhfm\" (UID: \"9e88beb3-f59b-4503-8215-a3e8f4cde9c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dmhfm"
Sep 09 18:35:53.944361 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:53.944320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e88beb3-f59b-4503-8215-a3e8f4cde9c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dmhfm\" (UID: \"9e88beb3-f59b-4503-8215-a3e8f4cde9c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dmhfm"
Sep 09 18:35:53.944548 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:53.944481    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 09 18:35:53.944594 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:35:53.944584    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e88beb3-f59b-4503-8215-a3e8f4cde9c0-tls-certificates podName:9e88beb3-f59b-4503-8215-a3e8f4cde9c0 nodeName:}" failed. No retries permitted until 2026-09-09 18:35:54.4445624 +0000 UTC m=+242.681932207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/9e88beb3-f59b-4503-8215-a3e8f4cde9c0-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-dmhfm" (UID: "9e88beb3-f59b-4503-8215-a3e8f4cde9c0") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 09 18:35:54.085085 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:54.084982    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl" event={"ID":"b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8","Type":"ContainerStarted","Data":"d805823ed14bb3092796a59de6140e86032e221bc7ec6e80298daae9c5e5b75c"}
Sep 09 18:35:54.101009 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:54.100956    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-7jqpl" podStartSLOduration=34.035661383 podStartE2EDuration="36.100927765s" totalImagesPullingTime="2.065266382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:35:54.099350028 +0000 UTC m=+242.336719844" watchObservedRunningTime="2026-09-09 18:35:54.100927765 +0000 UTC m=+242.338297581"
Sep 09 18:35:54.448470 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:54.448435    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e88beb3-f59b-4503-8215-a3e8f4cde9c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dmhfm\" (UID: \"9e88beb3-f59b-4503-8215-a3e8f4cde9c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dmhfm"
Sep 09 18:35:54.450938 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:54.450913    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e88beb3-f59b-4503-8215-a3e8f4cde9c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dmhfm\" (UID: \"9e88beb3-f59b-4503-8215-a3e8f4cde9c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dmhfm"
Sep 09 18:35:54.617251 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:54.617209    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dmhfm"
Sep 09 18:35:54.750593 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:54.750480    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:54.752966 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:54.752935    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tlfq9\" (UID: \"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3\") " pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:54.884882 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:54.884848    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dmhfm"]
Sep 09 18:35:54.887733 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:35:54.887698    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e88beb3_f59b_4503_8215_a3e8f4cde9c0.slice/crio-a2e96a23d8b315d747cfd58ceeb9d84f87e9fadd482aaca0b0aedb90a69d488b WatchSource:0}: Error finding container a2e96a23d8b315d747cfd58ceeb9d84f87e9fadd482aaca0b0aedb90a69d488b: Status 404 returned error can't find the container with id a2e96a23d8b315d747cfd58ceeb9d84f87e9fadd482aaca0b0aedb90a69d488b
Sep 09 18:35:54.974221 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:54.974189    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nn2fg\""
Sep 09 18:35:54.982185 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:54.982155    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tlfq9"
Sep 09 18:35:55.089283 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:55.089248    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dmhfm" event={"ID":"9e88beb3-f59b-4503-8215-a3e8f4cde9c0","Type":"ContainerStarted","Data":"a2e96a23d8b315d747cfd58ceeb9d84f87e9fadd482aaca0b0aedb90a69d488b"}
Sep 09 18:35:55.252337 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:55.252260    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tlfq9"]
Sep 09 18:35:55.255046 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:35:55.255015    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0bd9c1_3dd5_4b8b_b581_844fa31f94e3.slice/crio-e46aa48519878e73a9352536eaf042c60fc2f75fe3c1ee42e4bcb78db3c39589 WatchSource:0}: Error finding container e46aa48519878e73a9352536eaf042c60fc2f75fe3c1ee42e4bcb78db3c39589: Status 404 returned error can't find the container with id e46aa48519878e73a9352536eaf042c60fc2f75fe3c1ee42e4bcb78db3c39589
Sep 09 18:35:56.094132 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:56.094092    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlfq9" event={"ID":"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3","Type":"ContainerStarted","Data":"4ad86d70bcfaa95db5a5952080d997b4ccc052094772e58c4a7bb8d41e967363"}
Sep 09 18:35:56.094550 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:56.094182    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlfq9" event={"ID":"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3","Type":"ContainerStarted","Data":"e46aa48519878e73a9352536eaf042c60fc2f75fe3c1ee42e4bcb78db3c39589"}
Sep 09 18:35:57.099920 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:57.099883    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlfq9" event={"ID":"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3","Type":"ContainerStarted","Data":"e5503f907db4ab3e6310c99f07645a3f1b81294acae76374d3f10fe513ab3f9b"}
Sep 09 18:35:59.110050 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:59.110016    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dmhfm" event={"ID":"9e88beb3-f59b-4503-8215-a3e8f4cde9c0","Type":"ContainerStarted","Data":"e9b53c7410970888d3991814142244dc88bbe4c1688edd8e36cc59e9909f7379"}
Sep 09 18:35:59.110552 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:59.110182    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dmhfm"
Sep 09 18:35:59.114955 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:59.114928    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dmhfm"
Sep 09 18:35:59.124308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:35:59.124256    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dmhfm" podStartSLOduration=2.537965469 podStartE2EDuration="6.124239575s" totalImagesPullingTime="3.586274106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:35:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:35:59.123796974 +0000 UTC m=+247.361166792" watchObservedRunningTime="2026-09-09 18:35:59.124239575 +0000 UTC m=+247.361609393"
Sep 09 18:36:01.119035 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:01.118995    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tlfq9" event={"ID":"dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3","Type":"ContainerStarted","Data":"7816aa5cc1bc0dfa9a99f5b7f6969cc9c610500e1af6b942893708d5834a55af"}
Sep 09 18:36:01.136557 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:01.136472    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tlfq9" podStartSLOduration=34.011731284 podStartE2EDuration="39.136448562s" totalImagesPullingTime="5.124717278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 18:35:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:36:01.135900823 +0000 UTC m=+249.373270640" watchObservedRunningTime="2026-09-09 18:36:01.136448562 +0000 UTC m=+249.373818379"
Sep 09 18:36:01.314924 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:01.314881    2575 scope.go:117] "RemoveContainer" containerID="53d02c2203463f148b952b25d57ab7583d810b1aef4ceaf369e0a5fee7b1344e"
Sep 09 18:36:01.315090 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:36:01.315071    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-nmt4x_openshift-console-operator(b2848f11-65c4-4624-8cae-a9c7de831abf)\"" pod="openshift-console-operator/console-operator-85b454874b-nmt4x" podUID="b2848f11-65c4-4624-8cae-a9c7de831abf"
Sep 09 18:36:04.079011 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.078969    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"]
Sep 09 18:36:04.082170 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.082142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"
Sep 09 18:36:04.086104 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.085830    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 18:36:04.086104 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.085872    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 18:36:04.086104 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.085952    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 18:36:04.086104 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.085872    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-65mg9\""
Sep 09 18:36:04.093915 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.093888    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"]
Sep 09 18:36:04.118341 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.118291    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4c5fq"]
Sep 09 18:36:04.121079 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.121057    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.123413 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.123387    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 18:36:04.123562 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.123413    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rcsch\""
Sep 09 18:36:04.123841 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.123819    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 18:36:04.124158 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.124133    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvjr\" (UniqueName: \"kubernetes.io/projected/f9f44554-99ce-4fdf-9b59-7635b1cc48e7-kube-api-access-9jvjr\") pod \"openshift-state-metrics-657fd6c54-n6pjv\" (UID: \"f9f44554-99ce-4fdf-9b59-7635b1cc48e7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"
Sep 09 18:36:04.124240 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.124180    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-sys\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.124240 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.124216    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-root\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.124361 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.124245    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-textfile\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.124361 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.124274    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9f44554-99ce-4fdf-9b59-7635b1cc48e7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-n6pjv\" (UID: \"f9f44554-99ce-4fdf-9b59-7635b1cc48e7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"
Sep 09 18:36:04.124361 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.124309    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-accelerators-collector-config\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.124361 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.124335    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-wtmp\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.124361 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.124356    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-metrics-client-ca\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.124629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.124378    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd9fs\" (UniqueName: \"kubernetes.io/projected/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-kube-api-access-hd9fs\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.124629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.124439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9f44554-99ce-4fdf-9b59-7635b1cc48e7-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-n6pjv\" (UID: \"f9f44554-99ce-4fdf-9b59-7635b1cc48e7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"
Sep 09 18:36:04.124629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.124471    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-tls\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.124629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.124526    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:36:04.124629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.124566    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9f44554-99ce-4fdf-9b59-7635b1cc48e7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-n6pjv\" (UID: \"f9f44554-99ce-4fdf-9b59-7635b1cc48e7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"
Sep 09 18:36:04.124629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.124600    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.124907 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.123960    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 18:36:04.127878 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.127851    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a83d9228-1114-4294-b196-63f6bd1a0fdd-metrics-certs\") pod \"network-metrics-daemon-kdnq9\" (UID: \"a83d9228-1114-4294-b196-63f6bd1a0fdd\") " pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:36:04.224947 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.224894    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9f44554-99ce-4fdf-9b59-7635b1cc48e7-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-n6pjv\" (UID: \"f9f44554-99ce-4fdf-9b59-7635b1cc48e7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"
Sep 09 18:36:04.224947 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.224945    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-tls\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.225186 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.224981    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9f44554-99ce-4fdf-9b59-7635b1cc48e7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-n6pjv\" (UID: \"f9f44554-99ce-4fdf-9b59-7635b1cc48e7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"
Sep 09 18:36:04.225186 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.225009    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.225186 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.225041    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvjr\" (UniqueName: \"kubernetes.io/projected/f9f44554-99ce-4fdf-9b59-7635b1cc48e7-kube-api-access-9jvjr\") pod \"openshift-state-metrics-657fd6c54-n6pjv\" (UID: \"f9f44554-99ce-4fdf-9b59-7635b1cc48e7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"
Sep 09 18:36:04.225186 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:36:04.225142    2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 18:36:04.225490 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:36:04.225467    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-tls podName:3c9db26d-4df7-48e0-9a7c-eb664a8eeb53 nodeName:}" failed. No retries permitted until 2026-09-09 18:36:04.725439808 +0000 UTC m=+252.962809605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-tls") pod "node-exporter-4c5fq" (UID: "3c9db26d-4df7-48e0-9a7c-eb664a8eeb53") : secret "node-exporter-tls" not found
Sep 09 18:36:04.225702 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.225676    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-sys\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.225877 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.225852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-root\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.225991 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.225892    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-textfile\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.225991 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.225929    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9f44554-99ce-4fdf-9b59-7635b1cc48e7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-n6pjv\" (UID: \"f9f44554-99ce-4fdf-9b59-7635b1cc48e7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"
Sep 09 18:36:04.225991 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.225937    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-root\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.225991 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.225745    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9f44554-99ce-4fdf-9b59-7635b1cc48e7-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-n6pjv\" (UID: \"f9f44554-99ce-4fdf-9b59-7635b1cc48e7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"
Sep 09 18:36:04.225991 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.225968    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-accelerators-collector-config\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.226235 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.226022    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-wtmp\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.226235 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.226050    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-metrics-client-ca\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.226235 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.226074    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd9fs\" (UniqueName: \"kubernetes.io/projected/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-kube-api-access-hd9fs\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.226711 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.226458    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-wtmp\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.226711 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.225800    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-sys\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.226711 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.226565    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-accelerators-collector-config\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.226711 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.226668    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-textfile\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.227648 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.227624    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-metrics-client-ca\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.228091 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.228063    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.228253 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.228229    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9f44554-99ce-4fdf-9b59-7635b1cc48e7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-n6pjv\" (UID: \"f9f44554-99ce-4fdf-9b59-7635b1cc48e7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"
Sep 09 18:36:04.228970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.228949    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9f44554-99ce-4fdf-9b59-7635b1cc48e7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-n6pjv\" (UID: \"f9f44554-99ce-4fdf-9b59-7635b1cc48e7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"
Sep 09 18:36:04.233060 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.233035    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvjr\" (UniqueName: \"kubernetes.io/projected/f9f44554-99ce-4fdf-9b59-7635b1cc48e7-kube-api-access-9jvjr\") pod \"openshift-state-metrics-657fd6c54-n6pjv\" (UID: \"f9f44554-99ce-4fdf-9b59-7635b1cc48e7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"
Sep 09 18:36:04.233934 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.233915    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd9fs\" (UniqueName: \"kubernetes.io/projected/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-kube-api-access-hd9fs\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.318238 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.318204    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2784s\""
Sep 09 18:36:04.325986 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.325950    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kdnq9"
Sep 09 18:36:04.393798 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.393760    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"
Sep 09 18:36:04.603208 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.603122    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kdnq9"]
Sep 09 18:36:04.606818 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:36:04.606793    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83d9228_1114_4294_b196_63f6bd1a0fdd.slice/crio-2bd06f41503cc642e19e1b71ef6d1f7bb5fba8aea7e08c63fe96806567e42b6f WatchSource:0}: Error finding container 2bd06f41503cc642e19e1b71ef6d1f7bb5fba8aea7e08c63fe96806567e42b6f: Status 404 returned error can't find the container with id 2bd06f41503cc642e19e1b71ef6d1f7bb5fba8aea7e08c63fe96806567e42b6f
Sep 09 18:36:04.675597 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.675564    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv"]
Sep 09 18:36:04.678447 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:36:04.678414    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f44554_99ce_4fdf_9b59_7635b1cc48e7.slice/crio-06e08c2914e021d998c1963a51062ad2e93770c48bd4c7895172acc1ca4612fa WatchSource:0}: Error finding container 06e08c2914e021d998c1963a51062ad2e93770c48bd4c7895172acc1ca4612fa: Status 404 returned error can't find the container with id 06e08c2914e021d998c1963a51062ad2e93770c48bd4c7895172acc1ca4612fa
Sep 09 18:36:04.731256 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.731222    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-tls\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.733846 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.733815    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3c9db26d-4df7-48e0-9a7c-eb664a8eeb53-node-exporter-tls\") pod \"node-exporter-4c5fq\" (UID: \"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53\") " pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.739241 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:04.739223    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4c5fq"
Sep 09 18:36:04.753052 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:36:04.753009    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9db26d_4df7_48e0_9a7c_eb664a8eeb53.slice/crio-afa6e9016b13929f425a0fd2cc6b315490de0506becf0daf16ec6ec618d18494 WatchSource:0}: Error finding container afa6e9016b13929f425a0fd2cc6b315490de0506becf0daf16ec6ec618d18494: Status 404 returned error can't find the container with id afa6e9016b13929f425a0fd2cc6b315490de0506becf0daf16ec6ec618d18494
Sep 09 18:36:05.134826 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.134774    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4c5fq" event={"ID":"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53","Type":"ContainerStarted","Data":"afa6e9016b13929f425a0fd2cc6b315490de0506becf0daf16ec6ec618d18494"}
Sep 09 18:36:05.136916 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.136878    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv" event={"ID":"f9f44554-99ce-4fdf-9b59-7635b1cc48e7","Type":"ContainerStarted","Data":"83dcfc5d243cce2febce7de859505dd307e1c175de061c88cc6a72b09acc6b01"}
Sep 09 18:36:05.137064 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.136947    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv" event={"ID":"f9f44554-99ce-4fdf-9b59-7635b1cc48e7","Type":"ContainerStarted","Data":"51dae502546b19f1deeb9f734af95c149dc17263182b6c0c037abf3e3df7b089"}
Sep 09 18:36:05.137064 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.136977    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv" event={"ID":"f9f44554-99ce-4fdf-9b59-7635b1cc48e7","Type":"ContainerStarted","Data":"06e08c2914e021d998c1963a51062ad2e93770c48bd4c7895172acc1ca4612fa"}
Sep 09 18:36:05.138187 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.138146    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kdnq9" event={"ID":"a83d9228-1114-4294-b196-63f6bd1a0fdd","Type":"ContainerStarted","Data":"2bd06f41503cc642e19e1b71ef6d1f7bb5fba8aea7e08c63fe96806567e42b6f"}
Sep 09 18:36:05.147287 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.147256    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 18:36:05.150207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.150183    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.152647 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.152606    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 18:36:05.152790 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.152666    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 18:36:05.152790 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.152716    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 18:36:05.152790 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.152724    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 18:36:05.153127 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.153110    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 18:36:05.153354 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.153338    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 18:36:05.153543 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.153525    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 18:36:05.153627 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.153525    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 18:36:05.153627 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.153605    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 18:36:05.153742 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.153627    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gdr49\""
Sep 09 18:36:05.164709 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.164680    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 18:36:05.235649 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.235594    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-config-out\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.235649 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.235651    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.235925 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.235686    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.235925 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.235708    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.235925 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.235730    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.235925 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.235777    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.235925 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.235821    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.235925 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.235850    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzwxn\" (UniqueName: \"kubernetes.io/projected/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-kube-api-access-nzwxn\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.235925 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.235906    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.236273 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.235929    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.236273 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.235960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-config-volume\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.236273 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.235976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-web-config\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.236273 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.235992    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.336877 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.336831    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-config-out\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.337088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.336903    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.337088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.336945    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.337088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.336966    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.337088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.337001    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.337088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.337025    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.337088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.337055    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.337088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.337086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzwxn\" (UniqueName: \"kubernetes.io/projected/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-kube-api-access-nzwxn\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.337423 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.337132    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.337423 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.337200    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.337423 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.337245    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-config-volume\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.337423 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.337269    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-web-config\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.337423 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.337294    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.337694 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.337427    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.338700 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.338516    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.338919 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:36:05.338901    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-trusted-ca-bundle podName:cbf6cf70-9ee8-43e2-844d-b7ad093c2742 nodeName:}" failed. No retries permitted until 2026-09-09 18:36:05.837856945 +0000 UTC m=+254.075226755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "cbf6cf70-9ee8-43e2-844d-b7ad093c2742") : configmap references non-existent config key: ca-bundle.crt
Sep 09 18:36:05.340393 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.340369    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-web-config\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.341212 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.341130    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.341334 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.341215    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.342394 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.342299    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-config-out\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.342526 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.342393    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.342835 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.342815    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.342902 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.342846    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-config-volume\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.343078 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.343054    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.343357 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.343340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.350214 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.350182    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzwxn\" (UniqueName: \"kubernetes.io/projected/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-kube-api-access-nzwxn\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.841452 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.841291    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:05.842617 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:05.842432    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:06.061103 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:06.061060    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:36:06.785554 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:06.785518    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 18:36:06.788480 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:36:06.788450    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf6cf70_9ee8_43e2_844d_b7ad093c2742.slice/crio-71a7052363473856b3ebe8c1f4525d2a60a694baf431e1e38cef9cbb76777460 WatchSource:0}: Error finding container 71a7052363473856b3ebe8c1f4525d2a60a694baf431e1e38cef9cbb76777460: Status 404 returned error can't find the container with id 71a7052363473856b3ebe8c1f4525d2a60a694baf431e1e38cef9cbb76777460
Sep 09 18:36:07.151079 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:07.151044    2575 generic.go:356] "Generic (PLEG): container finished" podID="3c9db26d-4df7-48e0-9a7c-eb664a8eeb53" containerID="5d316de605818791d0801bc6c03b49476e99aa2305898c64aeca413529ab8c37" exitCode=0
Sep 09 18:36:07.151282 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:07.151130    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4c5fq" event={"ID":"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53","Type":"ContainerDied","Data":"5d316de605818791d0801bc6c03b49476e99aa2305898c64aeca413529ab8c37"}
Sep 09 18:36:07.153801 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:07.153766    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv" event={"ID":"f9f44554-99ce-4fdf-9b59-7635b1cc48e7","Type":"ContainerStarted","Data":"583b424f0726458503d40f9ba3188ec3840b796ab7a1d7f7aed358fe7a3b6de9"}
Sep 09 18:36:07.155990 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:07.155962    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kdnq9" event={"ID":"a83d9228-1114-4294-b196-63f6bd1a0fdd","Type":"ContainerStarted","Data":"3472cc64c9bb1ad2aa693c8740665df3fa9e007193cf47be6bdeaefae35cf4e6"}
Sep 09 18:36:07.156247 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:07.156026    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kdnq9" event={"ID":"a83d9228-1114-4294-b196-63f6bd1a0fdd","Type":"ContainerStarted","Data":"31b9617413a636124102b24d906f0912a75eae980f44fa4dab9d38026bc53e0a"}
Sep 09 18:36:07.157512 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:07.157471    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerStarted","Data":"71a7052363473856b3ebe8c1f4525d2a60a694baf431e1e38cef9cbb76777460"}
Sep 09 18:36:07.183227 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:07.183167    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kdnq9" podStartSLOduration=253.294135846 podStartE2EDuration="4m15.183149774s" totalImagesPullingTime="1.889013928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:36:07.182266363 +0000 UTC m=+255.419636191" watchObservedRunningTime="2026-09-09 18:36:07.183149774 +0000 UTC m=+255.420519588"
Sep 09 18:36:07.209760 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:07.206051    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-n6pjv" podStartSLOduration=1.512620451 podStartE2EDuration="3.206027496s" totalImagesPullingTime="1.693407045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:36:07.202098392 +0000 UTC m=+255.439468211" watchObservedRunningTime="2026-09-09 18:36:07.206027496 +0000 UTC m=+255.443397312"
Sep 09 18:36:08.163295 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.163253    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4c5fq" event={"ID":"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53","Type":"ContainerStarted","Data":"329e308ad48c6b4ebeb13a514c165b8f30445a13da5e28f41e9aa750563c886b"}
Sep 09 18:36:08.163736 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.163377    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4c5fq" event={"ID":"3c9db26d-4df7-48e0-9a7c-eb664a8eeb53","Type":"ContainerStarted","Data":"e84d098e14051a9e11ba371795d781d32e4a5c688218fda5aa6e0cb2f6bffabe"}
Sep 09 18:36:08.164941 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.164916    2575 generic.go:356] "Generic (PLEG): container finished" podID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerID="72a0ffbc8682eb642b5f4978f9ee8d85364b944399ea7e2587699e6bf58879c9" exitCode=0
Sep 09 18:36:08.165067 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.164955    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerDied","Data":"72a0ffbc8682eb642b5f4978f9ee8d85364b944399ea7e2587699e6bf58879c9"}
Sep 09 18:36:08.180468 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.180416    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4c5fq" podStartSLOduration=2.357487952 podStartE2EDuration="4.180397737s" totalImagesPullingTime="1.742355829s" totalInitContainerRuntime="80.553956ms" isStatefulPod=true podCreationTimestamp="2026-09-09 18:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:36:08.178913578 +0000 UTC m=+256.416283398" watchObservedRunningTime="2026-09-09 18:36:08.180397737 +0000 UTC m=+256.417767552"
Sep 09 18:36:08.391053 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.391016    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-64549dc5b-hczq5"]
Sep 09 18:36:08.393367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.393348    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.396030 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.396001    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 18:36:08.396155 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.396084    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 18:36:08.396285 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.396261    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bbojnk3jg49p8\""
Sep 09 18:36:08.396430 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.396294    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-j48cz\""
Sep 09 18:36:08.396430 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.396311    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 18:36:08.396610 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.396594    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 18:36:08.403554 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.403530    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64549dc5b-hczq5"]
Sep 09 18:36:08.464530 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.464461    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/78cad44e-5dd0-41be-9846-367f23eeb558-client-ca-bundle\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.464707 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.464548    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78cad44e-5dd0-41be-9846-367f23eeb558-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.464707 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.464593    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/78cad44e-5dd0-41be-9846-367f23eeb558-audit-log\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.464707 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.464621    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/78cad44e-5dd0-41be-9846-367f23eeb558-metrics-server-audit-profiles\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.464707 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.464685    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/78cad44e-5dd0-41be-9846-367f23eeb558-secret-metrics-server-client-certs\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.464889 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.464720    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smkxg\" (UniqueName: \"kubernetes.io/projected/78cad44e-5dd0-41be-9846-367f23eeb558-kube-api-access-smkxg\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.464889 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.464750    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/78cad44e-5dd0-41be-9846-367f23eeb558-secret-metrics-server-tls\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.565939 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.565902    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smkxg\" (UniqueName: \"kubernetes.io/projected/78cad44e-5dd0-41be-9846-367f23eeb558-kube-api-access-smkxg\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.566064 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.565946    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/78cad44e-5dd0-41be-9846-367f23eeb558-secret-metrics-server-tls\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.566064 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.565974    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/78cad44e-5dd0-41be-9846-367f23eeb558-client-ca-bundle\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.566064 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.565995    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78cad44e-5dd0-41be-9846-367f23eeb558-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.566064 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.566024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/78cad44e-5dd0-41be-9846-367f23eeb558-audit-log\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.566064 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.566041    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/78cad44e-5dd0-41be-9846-367f23eeb558-metrics-server-audit-profiles\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.566302 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.566086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/78cad44e-5dd0-41be-9846-367f23eeb558-secret-metrics-server-client-certs\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.566593 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.566560    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/78cad44e-5dd0-41be-9846-367f23eeb558-audit-log\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.566855 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.566829    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78cad44e-5dd0-41be-9846-367f23eeb558-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.567098 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.567077    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/78cad44e-5dd0-41be-9846-367f23eeb558-metrics-server-audit-profiles\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.568493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.568464    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/78cad44e-5dd0-41be-9846-367f23eeb558-client-ca-bundle\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.568605 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.568525    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/78cad44e-5dd0-41be-9846-367f23eeb558-secret-metrics-server-tls\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.568647 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.568611    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/78cad44e-5dd0-41be-9846-367f23eeb558-secret-metrics-server-client-certs\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.573594 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.573570    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smkxg\" (UniqueName: \"kubernetes.io/projected/78cad44e-5dd0-41be-9846-367f23eeb558-kube-api-access-smkxg\") pod \"metrics-server-64549dc5b-hczq5\" (UID: \"78cad44e-5dd0-41be-9846-367f23eeb558\") " pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.703041 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.702934    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:08.974142 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:08.974066    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64549dc5b-hczq5"]
Sep 09 18:36:08.977057 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:36:08.977030    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78cad44e_5dd0_41be_9846_367f23eeb558.slice/crio-9a8a6f00386212a667fc3b4329ff4f3e43e1e8941c205f222b9b1d153b34d69f WatchSource:0}: Error finding container 9a8a6f00386212a667fc3b4329ff4f3e43e1e8941c205f222b9b1d153b34d69f: Status 404 returned error can't find the container with id 9a8a6f00386212a667fc3b4329ff4f3e43e1e8941c205f222b9b1d153b34d69f
Sep 09 18:36:09.171026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:09.170989    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64549dc5b-hczq5" event={"ID":"78cad44e-5dd0-41be-9846-367f23eeb558","Type":"ContainerStarted","Data":"9a8a6f00386212a667fc3b4329ff4f3e43e1e8941c205f222b9b1d153b34d69f"}
Sep 09 18:36:10.178565 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.178494    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerStarted","Data":"6e4ee7a978d6d752ae4496e7f8cf1ae85429eae7740dd5894fd7fcd21ad1353b"}
Sep 09 18:36:10.269016 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.268917    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 18:36:10.272927 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.272893    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.275406 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.275378    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 18:36:10.275642 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.275542    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 18:36:10.275642 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.275573    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 18:36:10.275786 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.275398    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 18:36:10.275786 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.275734    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 18:36:10.275786 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.275389    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 18:36:10.276700 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.276650    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 18:36:10.276700 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.276686    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 18:36:10.276910 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.276800    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 18:36:10.277055 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.277036    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-pbhr060igvkh\""
Sep 09 18:36:10.277133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.277108    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 18:36:10.277133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.277122    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 18:36:10.277217 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.277160    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 18:36:10.277217 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.277039    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 18:36:10.282482 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.278172    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-flv6g\""
Sep 09 18:36:10.284865 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.284822    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 18:36:10.290441 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.290412    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 18:36:10.384344 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.384305    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.384532 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.384355    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.384532 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.384433    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.384532 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.384487    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.384690 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.384558    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.384690 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.384603    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.384690 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.384631    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.384822 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.384710    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdqz2\" (UniqueName: \"kubernetes.io/projected/b5009f37-b12d-4a23-af52-9372ca9b0ec4-kube-api-access-sdqz2\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.384822 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.384763    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.384822 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.384808    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.384949 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.384836    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.384949 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.384856    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.384949 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.384902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.385092 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.384982    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.385092 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.385014    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-config\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.385092 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.385055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.385236 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.385111    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5009f37-b12d-4a23-af52-9372ca9b0ec4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.385236 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.385146    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.385236 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.385211    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.385365 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.385270    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5009f37-b12d-4a23-af52-9372ca9b0ec4-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.485780 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.485750    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.485899 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.485797    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.485899 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.485826    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.485899 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.485883    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.486081 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.485927    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-config\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.486081 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.485959    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.487190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.486013    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5009f37-b12d-4a23-af52-9372ca9b0ec4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.487190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.486473    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.487190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.486557    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.487190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.486592    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5009f37-b12d-4a23-af52-9372ca9b0ec4-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.487190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.486614    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.487190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.486613    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.487190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.486638    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.487190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.486924    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.487190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.487045    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.489759 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.488882    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.489759 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.489060    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-config\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.489759 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.489684    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.489759 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.489744    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.490009 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.489778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.490009 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.489812    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.490009 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.489838    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.490009 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.489866    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.490009 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.489911    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdqz2\" (UniqueName: \"kubernetes.io/projected/b5009f37-b12d-4a23-af52-9372ca9b0ec4-kube-api-access-sdqz2\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.490009 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.489962    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.490009 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.490007    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.490328 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.490023    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.490328 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.490141    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.490740 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.490718    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.490740 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.490732    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.491801 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.491339    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5009f37-b12d-4a23-af52-9372ca9b0ec4-config-out\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.491801 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.491485    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.491801 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.491666    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5009f37-b12d-4a23-af52-9372ca9b0ec4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.492347 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.492321    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.492479 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.492451    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.493931 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.493889    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-web-config\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.493931 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.493889    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.494640 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.494615    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.495077 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.495057    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.498825 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.498799    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdqz2\" (UniqueName: \"kubernetes.io/projected/b5009f37-b12d-4a23-af52-9372ca9b0ec4-kube-api-access-sdqz2\") pod \"prometheus-k8s-0\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.581525 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.581451    2575 patch_prober.go:28] interesting pod/image-registry-6b7bd5bbf4-6sctb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 18:36:10.581722 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.581535    2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb" podUID="7b27f60e-eb9e-41f5-8141-6c2ed0448395" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 18:36:10.591118 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.591079    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:10.875459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:10.875292    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 18:36:10.878066 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:36:10.878032    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5009f37_b12d_4a23_af52_9372ca9b0ec4.slice/crio-384e0db2fad6bcf6e71d5fa12c375be5fce5cf5d6556cf40dc2d619d194cfc21 WatchSource:0}: Error finding container 384e0db2fad6bcf6e71d5fa12c375be5fce5cf5d6556cf40dc2d619d194cfc21: Status 404 returned error can't find the container with id 384e0db2fad6bcf6e71d5fa12c375be5fce5cf5d6556cf40dc2d619d194cfc21
Sep 09 18:36:11.184436 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:11.184344    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64549dc5b-hczq5" event={"ID":"78cad44e-5dd0-41be-9846-367f23eeb558","Type":"ContainerStarted","Data":"a0ad62aa9232af9c4cee22d3085aacf4699bb5b81036bc61dae112c88c305792"}
Sep 09 18:36:11.186063 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:11.186033    2575 generic.go:356] "Generic (PLEG): container finished" podID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerID="63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77" exitCode=0
Sep 09 18:36:11.186169 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:11.186123    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerDied","Data":"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"}
Sep 09 18:36:11.186253 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:11.186206    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerStarted","Data":"384e0db2fad6bcf6e71d5fa12c375be5fce5cf5d6556cf40dc2d619d194cfc21"}
Sep 09 18:36:11.190058 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:11.189948    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerStarted","Data":"d748799378eb040a61d7572f8adaa80774decd033d9279e21c3ae6bd218e04ea"}
Sep 09 18:36:11.190178 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:11.190090    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerStarted","Data":"6b7d57e67746ffc363dbbcfce3b3e2953f33b7e9878fa989dd6a2e5dfde2e8cc"}
Sep 09 18:36:11.190178 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:11.190133    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerStarted","Data":"6ab4857798adeaffe6f28b284452bd10cc2d7471caa2262e52dbe978858fa4ee"}
Sep 09 18:36:11.190309 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:11.190182    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerStarted","Data":"9780cc34d27a6556a79346c166369af84690e598e056a951dbd99b7ebf1a932e"}
Sep 09 18:36:11.205852 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:11.205796    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-64549dc5b-hczq5" podStartSLOduration=1.72983037 podStartE2EDuration="3.205780482s" totalImagesPullingTime="1.475950112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:36:11.204375704 +0000 UTC m=+259.441745518" watchObservedRunningTime="2026-09-09 18:36:11.205780482 +0000 UTC m=+259.443150298"
Sep 09 18:36:12.080856 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:12.080807    2575 patch_prober.go:28] interesting pod/image-registry-6b7bd5bbf4-6sctb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 18:36:12.081023 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:12.080876    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb" podUID="7b27f60e-eb9e-41f5-8141-6c2ed0448395" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 18:36:13.204352 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:13.204312    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerStarted","Data":"f4b57d0625032c2d41ea9d9f76d0cec6e29f5539a1bbdaedc95c2194240e7871"}
Sep 09 18:36:13.230052 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:13.229997    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.870328044 podStartE2EDuration="8.229978111s" totalImagesPullingTime="5.343472566s" totalInitContainerRuntime="16.177501ms" isStatefulPod=false podCreationTimestamp="2026-09-09 18:36:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:36:13.227824183 +0000 UTC m=+261.465194000" watchObservedRunningTime="2026-09-09 18:36:13.229978111 +0000 UTC m=+261.467347927"
Sep 09 18:36:15.216167 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:15.216112    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerStarted","Data":"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"}
Sep 09 18:36:15.216609 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:15.216191    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerStarted","Data":"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"}
Sep 09 18:36:15.315669 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:15.315631    2575 scope.go:117] "RemoveContainer" containerID="53d02c2203463f148b952b25d57ab7583d810b1aef4ceaf369e0a5fee7b1344e"
Sep 09 18:36:16.223668 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:16.223636    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 18:36:16.224153 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:16.223771    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-nmt4x" event={"ID":"b2848f11-65c4-4624-8cae-a9c7de831abf","Type":"ContainerStarted","Data":"7ca56ceb693e49181698063dfecc01d93d4ac71095238de9567c58b32d469c13"}
Sep 09 18:36:16.224235 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:16.224166    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:36:16.230489 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:16.230446    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-nmt4x"
Sep 09 18:36:16.242136 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:16.242058    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-nmt4x" podStartSLOduration=55.510058887 podStartE2EDuration="58.242035213s" totalImagesPullingTime="2.731976326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:36:16.238573702 +0000 UTC m=+264.475943530" watchObservedRunningTime="2026-09-09 18:36:16.242035213 +0000 UTC m=+264.479405030"
Sep 09 18:36:17.231556 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:17.231516    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerStarted","Data":"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"}
Sep 09 18:36:17.232020 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:17.231588    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerStarted","Data":"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"}
Sep 09 18:36:17.232020 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:17.231630    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerStarted","Data":"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"}
Sep 09 18:36:17.232020 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:17.231657    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerStarted","Data":"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"}
Sep 09 18:36:17.259012 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:17.258941    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.253379529 podStartE2EDuration="7.258915267s" totalImagesPullingTime="4.985714448s" totalInitContainerRuntime="19.82129ms" isStatefulPod=false podCreationTimestamp="2026-09-09 18:36:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:36:17.257193538 +0000 UTC m=+265.494563354" watchObservedRunningTime="2026-09-09 18:36:17.258915267 +0000 UTC m=+265.496285083"
Sep 09 18:36:20.580491 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:20.580452    2575 patch_prober.go:28] interesting pod/image-registry-6b7bd5bbf4-6sctb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 18:36:20.580916 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:20.580527    2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb" podUID="7b27f60e-eb9e-41f5-8141-6c2ed0448395" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 18:36:20.591287 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:20.591259    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:36:22.080643 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:22.080606    2575 patch_prober.go:28] interesting pod/image-registry-6b7bd5bbf4-6sctb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 18:36:22.081093 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:22.080678    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb" podUID="7b27f60e-eb9e-41f5-8141-6c2ed0448395" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 18:36:28.704040 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:28.704002    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:28.704443 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:28.704064    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:30.580541 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:30.580486    2575 patch_prober.go:28] interesting pod/image-registry-6b7bd5bbf4-6sctb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 18:36:30.581024 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:30.580573    2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb" podUID="7b27f60e-eb9e-41f5-8141-6c2ed0448395" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 18:36:30.581024 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:30.580623    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:36:30.581234 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:30.581181    2575 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"797719c9a4f462afa16e0427e5c903deacc03246b04c9abbe30dfda70186bd6e"} pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb" containerMessage="Container registry failed liveness probe, will be restarted"
Sep 09 18:36:30.584847 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:30.584818    2575 patch_prober.go:28] interesting pod/image-registry-6b7bd5bbf4-6sctb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 09 18:36:30.584983 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:30.584878    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb" podUID="7b27f60e-eb9e-41f5-8141-6c2ed0448395" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 18:36:35.827235 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:36:35.827179    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-lpmq9" podUID="4c571b1e-36e0-4647-a241-511ce5cfa737"
Sep 09 18:36:36.312240 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:36.312206    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:36:39.063172 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:39.063119    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:36:39.065596 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:39.065571    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4c571b1e-36e0-4647-a241-511ce5cfa737-metrics-tls\") pod \"dns-default-lpmq9\" (UID: \"4c571b1e-36e0-4647-a241-511ce5cfa737\") " pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:36:39.316095 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:39.316001    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dgjhb\""
Sep 09 18:36:39.323382 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:39.323344    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:36:39.600701 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:39.600674    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lpmq9"]
Sep 09 18:36:39.603338 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:36:39.603295    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c571b1e_36e0_4647_a241_511ce5cfa737.slice/crio-1297cb7a10af98975b7cc808ed0fd9a981da408e8acdadc6ada8994c05d9edf9 WatchSource:0}: Error finding container 1297cb7a10af98975b7cc808ed0fd9a981da408e8acdadc6ada8994c05d9edf9: Status 404 returned error can't find the container with id 1297cb7a10af98975b7cc808ed0fd9a981da408e8acdadc6ada8994c05d9edf9
Sep 09 18:36:40.330340 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:40.330298    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lpmq9" event={"ID":"4c571b1e-36e0-4647-a241-511ce5cfa737","Type":"ContainerStarted","Data":"1297cb7a10af98975b7cc808ed0fd9a981da408e8acdadc6ada8994c05d9edf9"}
Sep 09 18:36:40.586098 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:40.586015    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:36:42.340814 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:42.340708    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lpmq9" event={"ID":"4c571b1e-36e0-4647-a241-511ce5cfa737","Type":"ContainerStarted","Data":"f964d5e86afb82d4b31b9cb5601b248515a7394c858ddeb53b1d0ad827414c5e"}
Sep 09 18:36:42.341198 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:42.340876    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:36:42.341198 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:42.340916    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lpmq9" event={"ID":"4c571b1e-36e0-4647-a241-511ce5cfa737","Type":"ContainerStarted","Data":"72e0a89ba2dc0774bc3f0b8e6714bf93dff702451b4333b8aba984870d7c6fce"}
Sep 09 18:36:42.355635 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:42.355571    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lpmq9" podStartSLOduration=251.324337573 podStartE2EDuration="4m13.355547441s" totalImagesPullingTime="2.031209868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:36:42.355233058 +0000 UTC m=+290.592602875" watchObservedRunningTime="2026-09-09 18:36:42.355547441 +0000 UTC m=+290.592917257"
Sep 09 18:36:43.346647 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:43.346613    2575 generic.go:356] "Generic (PLEG): container finished" podID="f82752ac-77f8-44e5-a619-d1e1008923ce" containerID="e3576ee5ac68e517ce865114bc0e7d55e3a67a9e974551b8b421794175f35aa0" exitCode=0
Sep 09 18:36:43.347042 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:43.346694    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz" event={"ID":"f82752ac-77f8-44e5-a619-d1e1008923ce","Type":"ContainerDied","Data":"e3576ee5ac68e517ce865114bc0e7d55e3a67a9e974551b8b421794175f35aa0"}
Sep 09 18:36:43.347169 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:43.347155    2575 scope.go:117] "RemoveContainer" containerID="e3576ee5ac68e517ce865114bc0e7d55e3a67a9e974551b8b421794175f35aa0"
Sep 09 18:36:44.352654 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:44.352612    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-vz6kz" event={"ID":"f82752ac-77f8-44e5-a619-d1e1008923ce","Type":"ContainerStarted","Data":"21195976d360c0fb0628335954a632ad136d284e38fad5d4d80846a45053557a"}
Sep 09 18:36:46.300906 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:46.300878    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cbf6cf70-9ee8-43e2-844d-b7ad093c2742/init-config-reloader/0.log"
Sep 09 18:36:46.502839 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:46.502801    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cbf6cf70-9ee8-43e2-844d-b7ad093c2742/alertmanager/0.log"
Sep 09 18:36:46.701673 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:46.701643    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cbf6cf70-9ee8-43e2-844d-b7ad093c2742/config-reloader/0.log"
Sep 09 18:36:46.901174 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:46.901141    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cbf6cf70-9ee8-43e2-844d-b7ad093c2742/kube-rbac-proxy-web/0.log"
Sep 09 18:36:47.106914 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:47.106840    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cbf6cf70-9ee8-43e2-844d-b7ad093c2742/kube-rbac-proxy/0.log"
Sep 09 18:36:47.301037 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:47.301006    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cbf6cf70-9ee8-43e2-844d-b7ad093c2742/kube-rbac-proxy-metric/0.log"
Sep 09 18:36:47.501051 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:47.501014    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cbf6cf70-9ee8-43e2-844d-b7ad093c2742/prom-label-proxy/0.log"
Sep 09 18:36:47.704325 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:47.704293    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-7jqpl_b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8/cluster-monitoring-operator/0.log"
Sep 09 18:36:48.501553 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:48.501525    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-64549dc5b-hczq5_78cad44e-5dd0-41be-9846-367f23eeb558/metrics-server/0.log"
Sep 09 18:36:48.709794 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:48.709762    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:48.715245 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:48.715211    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-64549dc5b-hczq5"
Sep 09 18:36:48.901228 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:48.901201    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4c5fq_3c9db26d-4df7-48e0-9a7c-eb664a8eeb53/init-textfile/0.log"
Sep 09 18:36:49.101376 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:49.101346    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4c5fq_3c9db26d-4df7-48e0-9a7c-eb664a8eeb53/node-exporter/0.log"
Sep 09 18:36:49.301162 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:49.301084    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4c5fq_3c9db26d-4df7-48e0-9a7c-eb664a8eeb53/kube-rbac-proxy/0.log"
Sep 09 18:36:50.701156 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:50.701110    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-n6pjv_f9f44554-99ce-4fdf-9b59-7635b1cc48e7/kube-rbac-proxy-main/0.log"
Sep 09 18:36:50.900008 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:50.899974    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-n6pjv_f9f44554-99ce-4fdf-9b59-7635b1cc48e7/kube-rbac-proxy-self/0.log"
Sep 09 18:36:51.101602 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:51.101492    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-n6pjv_f9f44554-99ce-4fdf-9b59-7635b1cc48e7/openshift-state-metrics/0.log"
Sep 09 18:36:51.303996 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:51.303971    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b5009f37-b12d-4a23-af52-9372ca9b0ec4/init-config-reloader/0.log"
Sep 09 18:36:51.502524 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:51.502465    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b5009f37-b12d-4a23-af52-9372ca9b0ec4/prometheus/0.log"
Sep 09 18:36:51.701308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:51.701280    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b5009f37-b12d-4a23-af52-9372ca9b0ec4/config-reloader/0.log"
Sep 09 18:36:51.901096 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:51.901069    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b5009f37-b12d-4a23-af52-9372ca9b0ec4/thanos-sidecar/0.log"
Sep 09 18:36:52.105597 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:52.105572    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b5009f37-b12d-4a23-af52-9372ca9b0ec4/kube-rbac-proxy-web/0.log"
Sep 09 18:36:52.196162 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:52.196077    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 18:36:52.196335 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:52.196294    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 18:36:52.207362 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:52.207335    2575 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 18:36:52.305942 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:52.305777    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b5009f37-b12d-4a23-af52-9372ca9b0ec4/kube-rbac-proxy/0.log"
Sep 09 18:36:52.349444 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:52.349414    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lpmq9"
Sep 09 18:36:52.501063 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:52.500992    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b5009f37-b12d-4a23-af52-9372ca9b0ec4/kube-rbac-proxy-thanos/0.log"
Sep 09 18:36:53.101227 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:53.101199    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-dmhfm_9e88beb3-f59b-4503-8215-a3e8f4cde9c0/prometheus-operator-admission-webhook/0.log"
Sep 09 18:36:54.700251 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:54.700220    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 18:36:54.901690 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:54.901659    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/3.log"
Sep 09 18:36:55.601035 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:55.600985    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb" podUID="7b27f60e-eb9e-41f5-8141-6c2ed0448395" containerName="registry" containerID="cri-o://797719c9a4f462afa16e0427e5c903deacc03246b04c9abbe30dfda70186bd6e" gracePeriod=30
Sep 09 18:36:56.104130 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:56.104101    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lpmq9_4c571b1e-36e0-4647-a241-511ce5cfa737/dns/0.log"
Sep 09 18:36:56.301547 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:56.301492    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lpmq9_4c571b1e-36e0-4647-a241-511ce5cfa737/kube-rbac-proxy/0.log"
Sep 09 18:36:57.101027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:57.100950    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lpvdl_d4099f20-afa3-482b-a22e-c4d59554f7fd/dns-node-resolver/0.log"
Sep 09 18:36:57.415687 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:57.415655    2575 generic.go:356] "Generic (PLEG): container finished" podID="7b27f60e-eb9e-41f5-8141-6c2ed0448395" containerID="797719c9a4f462afa16e0427e5c903deacc03246b04c9abbe30dfda70186bd6e" exitCode=0
Sep 09 18:36:57.416137 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:57.415718    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb" event={"ID":"7b27f60e-eb9e-41f5-8141-6c2ed0448395","Type":"ContainerDied","Data":"797719c9a4f462afa16e0427e5c903deacc03246b04c9abbe30dfda70186bd6e"}
Sep 09 18:36:57.416137 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:57.415752    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb" event={"ID":"7b27f60e-eb9e-41f5-8141-6c2ed0448395","Type":"ContainerStarted","Data":"e4827eda2fc08d1eb2bc797c1885557f68ea8c1c3b5b0268bd3140110a9b5949"}
Sep 09 18:36:57.416137 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:57.415856    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:36:58.100720 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:58.100690    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xsjjs_d7b9ed88-9b45-457d-9e8e-9863f49d267b/serve-healthcheck-canary/0.log"
Sep 09 18:36:58.301784 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:58.301753    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-jnkr9_9598d5e4-eb11-4747-b7e7-c1b9681161b3/cluster-samples-operator/0.log"
Sep 09 18:36:58.502747 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:36:58.502720    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-jnkr9_9598d5e4-eb11-4747-b7e7-c1b9681161b3/cluster-samples-operator-watch/0.log"
Sep 09 18:37:10.591961 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:10.591924    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:10.612225 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:10.612198    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:11.490757 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:11.490729    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:18.424896 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:18.424865    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b7bd5bbf4-6sctb"
Sep 09 18:37:24.300747 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.300707    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 18:37:24.301277 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.301070    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="alertmanager" containerID="cri-o://6e4ee7a978d6d752ae4496e7f8cf1ae85429eae7740dd5894fd7fcd21ad1353b" gracePeriod=120
Sep 09 18:37:24.301277 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.301134    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="kube-rbac-proxy-web" containerID="cri-o://6ab4857798adeaffe6f28b284452bd10cc2d7471caa2262e52dbe978858fa4ee" gracePeriod=120
Sep 09 18:37:24.301277 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.301163    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="kube-rbac-proxy" containerID="cri-o://6b7d57e67746ffc363dbbcfce3b3e2953f33b7e9878fa989dd6a2e5dfde2e8cc" gracePeriod=120
Sep 09 18:37:24.301277 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.301196    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="config-reloader" containerID="cri-o://9780cc34d27a6556a79346c166369af84690e598e056a951dbd99b7ebf1a932e" gracePeriod=120
Sep 09 18:37:24.301277 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.301269    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="kube-rbac-proxy-metric" containerID="cri-o://d748799378eb040a61d7572f8adaa80774decd033d9279e21c3ae6bd218e04ea" gracePeriod=120
Sep 09 18:37:24.301561 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.301317    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="prom-label-proxy" containerID="cri-o://f4b57d0625032c2d41ea9d9f76d0cec6e29f5539a1bbdaedc95c2194240e7871" gracePeriod=120
Sep 09 18:37:24.531074 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.531035    2575 generic.go:356] "Generic (PLEG): container finished" podID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerID="f4b57d0625032c2d41ea9d9f76d0cec6e29f5539a1bbdaedc95c2194240e7871" exitCode=0
Sep 09 18:37:24.531074 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.531063    2575 generic.go:356] "Generic (PLEG): container finished" podID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerID="6b7d57e67746ffc363dbbcfce3b3e2953f33b7e9878fa989dd6a2e5dfde2e8cc" exitCode=0
Sep 09 18:37:24.531074 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.531072    2575 generic.go:356] "Generic (PLEG): container finished" podID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerID="9780cc34d27a6556a79346c166369af84690e598e056a951dbd99b7ebf1a932e" exitCode=0
Sep 09 18:37:24.531074 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.531080    2575 generic.go:356] "Generic (PLEG): container finished" podID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerID="6e4ee7a978d6d752ae4496e7f8cf1ae85429eae7740dd5894fd7fcd21ad1353b" exitCode=0
Sep 09 18:37:24.531351 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.531108    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerDied","Data":"f4b57d0625032c2d41ea9d9f76d0cec6e29f5539a1bbdaedc95c2194240e7871"}
Sep 09 18:37:24.531351 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.531172    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerDied","Data":"6b7d57e67746ffc363dbbcfce3b3e2953f33b7e9878fa989dd6a2e5dfde2e8cc"}
Sep 09 18:37:24.531351 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.531200    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerDied","Data":"9780cc34d27a6556a79346c166369af84690e598e056a951dbd99b7ebf1a932e"}
Sep 09 18:37:24.531351 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:24.531225    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerDied","Data":"6e4ee7a978d6d752ae4496e7f8cf1ae85429eae7740dd5894fd7fcd21ad1353b"}
Sep 09 18:37:25.539471 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.539440    2575 generic.go:356] "Generic (PLEG): container finished" podID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerID="d748799378eb040a61d7572f8adaa80774decd033d9279e21c3ae6bd218e04ea" exitCode=0
Sep 09 18:37:25.539471 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.539463    2575 generic.go:356] "Generic (PLEG): container finished" podID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerID="6ab4857798adeaffe6f28b284452bd10cc2d7471caa2262e52dbe978858fa4ee" exitCode=0
Sep 09 18:37:25.540028 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.539549    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerDied","Data":"d748799378eb040a61d7572f8adaa80774decd033d9279e21c3ae6bd218e04ea"}
Sep 09 18:37:25.540028 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.539618    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerDied","Data":"6ab4857798adeaffe6f28b284452bd10cc2d7471caa2262e52dbe978858fa4ee"}
Sep 09 18:37:25.555991 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.555962    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:25.681228 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681130    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-cluster-tls-config\") pod \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") "
Sep 09 18:37:25.681228 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681190    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-metrics-client-ca\") pod \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") "
Sep 09 18:37:25.681436 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681259    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy-metric\") pod \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") "
Sep 09 18:37:25.681436 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681305    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-main-db\") pod \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") "
Sep 09 18:37:25.681436 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681413    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-config-out\") pod \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") "
Sep 09 18:37:25.681618 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681490    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy\") pod \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") "
Sep 09 18:37:25.681618 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681552    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-tls-assets\") pod \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") "
Sep 09 18:37:25.681618 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681600    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "cbf6cf70-9ee8-43e2-844d-b7ad093c2742" (UID: "cbf6cf70-9ee8-43e2-844d-b7ad093c2742"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:37:25.681618 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681612    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-web-config\") pod \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") "
Sep 09 18:37:25.681830 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681626    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "cbf6cf70-9ee8-43e2-844d-b7ad093c2742" (UID: "cbf6cf70-9ee8-43e2-844d-b7ad093c2742"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:37:25.681830 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681652    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-main-tls\") pod \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") "
Sep 09 18:37:25.681830 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681685    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzwxn\" (UniqueName: \"kubernetes.io/projected/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-kube-api-access-nzwxn\") pod \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") "
Sep 09 18:37:25.681830 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681720    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy-web\") pod \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") "
Sep 09 18:37:25.681830 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681745    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-config-volume\") pod \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") "
Sep 09 18:37:25.681830 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.681786    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-trusted-ca-bundle\") pod \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\" (UID: \"cbf6cf70-9ee8-43e2-844d-b7ad093c2742\") "
Sep 09 18:37:25.682119 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.682068    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-metrics-client-ca\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:25.682119 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.682088    2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-main-db\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:25.682791 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.682527    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "cbf6cf70-9ee8-43e2-844d-b7ad093c2742" (UID: "cbf6cf70-9ee8-43e2-844d-b7ad093c2742"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:37:25.684481 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.684446    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "cbf6cf70-9ee8-43e2-844d-b7ad093c2742" (UID: "cbf6cf70-9ee8-43e2-844d-b7ad093c2742"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:25.684789 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.684738    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "cbf6cf70-9ee8-43e2-844d-b7ad093c2742" (UID: "cbf6cf70-9ee8-43e2-844d-b7ad093c2742"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:25.684902 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.684852    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-config-out" (OuterVolumeSpecName: "config-out") pod "cbf6cf70-9ee8-43e2-844d-b7ad093c2742" (UID: "cbf6cf70-9ee8-43e2-844d-b7ad093c2742"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:37:25.685196 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.685169    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "cbf6cf70-9ee8-43e2-844d-b7ad093c2742" (UID: "cbf6cf70-9ee8-43e2-844d-b7ad093c2742"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:37:25.685291 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.685270    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "cbf6cf70-9ee8-43e2-844d-b7ad093c2742" (UID: "cbf6cf70-9ee8-43e2-844d-b7ad093c2742"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:25.685434 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.685414    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-kube-api-access-nzwxn" (OuterVolumeSpecName: "kube-api-access-nzwxn") pod "cbf6cf70-9ee8-43e2-844d-b7ad093c2742" (UID: "cbf6cf70-9ee8-43e2-844d-b7ad093c2742"). InnerVolumeSpecName "kube-api-access-nzwxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:37:25.685946 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.685918    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "cbf6cf70-9ee8-43e2-844d-b7ad093c2742" (UID: "cbf6cf70-9ee8-43e2-844d-b7ad093c2742"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:25.686406 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.686380    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-config-volume" (OuterVolumeSpecName: "config-volume") pod "cbf6cf70-9ee8-43e2-844d-b7ad093c2742" (UID: "cbf6cf70-9ee8-43e2-844d-b7ad093c2742"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:25.689616 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.689475    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "cbf6cf70-9ee8-43e2-844d-b7ad093c2742" (UID: "cbf6cf70-9ee8-43e2-844d-b7ad093c2742"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:25.696972 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.696943    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-web-config" (OuterVolumeSpecName: "web-config") pod "cbf6cf70-9ee8-43e2-844d-b7ad093c2742" (UID: "cbf6cf70-9ee8-43e2-844d-b7ad093c2742"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:25.783100 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.783062    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-main-tls\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:25.783100 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.783094    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzwxn\" (UniqueName: \"kubernetes.io/projected/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-kube-api-access-nzwxn\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:25.783100 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.783105    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:25.783345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.783117    2575 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:25.783345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.783126    2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:25.783345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.783134    2575 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-cluster-tls-config\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:25.783345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.783142    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:25.783345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.783151    2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-config-out\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:25.783345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.783159    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:25.783345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.783167    2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-tls-assets\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:25.783345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:25.783176    2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cbf6cf70-9ee8-43e2-844d-b7ad093c2742-web-config\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:26.547148 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.547114    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cbf6cf70-9ee8-43e2-844d-b7ad093c2742","Type":"ContainerDied","Data":"71a7052363473856b3ebe8c1f4525d2a60a694baf431e1e38cef9cbb76777460"}
Sep 09 18:37:26.547655 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.547206    2575 scope.go:117] "RemoveContainer" containerID="f4b57d0625032c2d41ea9d9f76d0cec6e29f5539a1bbdaedc95c2194240e7871"
Sep 09 18:37:26.547655 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.547121    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.555354 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.555333    2575 scope.go:117] "RemoveContainer" containerID="d748799378eb040a61d7572f8adaa80774decd033d9279e21c3ae6bd218e04ea"
Sep 09 18:37:26.563229 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.563205    2575 scope.go:117] "RemoveContainer" containerID="6b7d57e67746ffc363dbbcfce3b3e2953f33b7e9878fa989dd6a2e5dfde2e8cc"
Sep 09 18:37:26.568551 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.568490    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 18:37:26.571878 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.571848    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 18:37:26.572677 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.572656    2575 scope.go:117] "RemoveContainer" containerID="6ab4857798adeaffe6f28b284452bd10cc2d7471caa2262e52dbe978858fa4ee"
Sep 09 18:37:26.580247 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.580227    2575 scope.go:117] "RemoveContainer" containerID="9780cc34d27a6556a79346c166369af84690e598e056a951dbd99b7ebf1a932e"
Sep 09 18:37:26.588459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.588438    2575 scope.go:117] "RemoveContainer" containerID="6e4ee7a978d6d752ae4496e7f8cf1ae85429eae7740dd5894fd7fcd21ad1353b"
Sep 09 18:37:26.595004 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.594978    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 18:37:26.595288 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595276    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="alertmanager"
Sep 09 18:37:26.595331 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595290    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="alertmanager"
Sep 09 18:37:26.595331 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595304    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="kube-rbac-proxy-metric"
Sep 09 18:37:26.595331 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595310    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="kube-rbac-proxy-metric"
Sep 09 18:37:26.595331 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595317    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="prom-label-proxy"
Sep 09 18:37:26.595331 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595322    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="prom-label-proxy"
Sep 09 18:37:26.595331 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595330    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="kube-rbac-proxy-web"
Sep 09 18:37:26.595493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595336    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="kube-rbac-proxy-web"
Sep 09 18:37:26.595493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595347    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="kube-rbac-proxy"
Sep 09 18:37:26.595493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595352    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="kube-rbac-proxy"
Sep 09 18:37:26.595493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595360    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="init-config-reloader"
Sep 09 18:37:26.595493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595365    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="init-config-reloader"
Sep 09 18:37:26.595493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595377    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="config-reloader"
Sep 09 18:37:26.595493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595385    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="config-reloader"
Sep 09 18:37:26.595493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595437    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="kube-rbac-proxy"
Sep 09 18:37:26.595493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595445    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="prom-label-proxy"
Sep 09 18:37:26.595493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595451    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="config-reloader"
Sep 09 18:37:26.595493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595457    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="kube-rbac-proxy-web"
Sep 09 18:37:26.595493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595463    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="alertmanager"
Sep 09 18:37:26.595493 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.595468    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" containerName="kube-rbac-proxy-metric"
Sep 09 18:37:26.597265 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.597241    2575 scope.go:117] "RemoveContainer" containerID="72a0ffbc8682eb642b5f4978f9ee8d85364b944399ea7e2587699e6bf58879c9"
Sep 09 18:37:26.599418 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.599398    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.601796 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.601777    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 18:37:26.601990 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.601972    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 18:37:26.602322 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.602302    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 18:37:26.602420 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.602354    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 18:37:26.602534 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.602515    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 18:37:26.602693 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.602618    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gdr49\""
Sep 09 18:37:26.602693 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.602646    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 18:37:26.602693 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.602656    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 18:37:26.602986 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.602719    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 18:37:26.610917 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.610883    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 18:37:26.616904 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.616870    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 18:37:26.691932 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.691882    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.692121 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.691947    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.692121 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.691989    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eea30ae-084d-4eb7-9b57-08d350bab832-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.692121 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.692008    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-config-volume\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.692121 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.692061    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4eea30ae-084d-4eb7-9b57-08d350bab832-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.692121 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.692088    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-web-config\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.692121 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.692108    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eea30ae-084d-4eb7-9b57-08d350bab832-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.692390 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.692126    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84xng\" (UniqueName: \"kubernetes.io/projected/4eea30ae-084d-4eb7-9b57-08d350bab832-kube-api-access-84xng\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.692390 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.692145    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4eea30ae-084d-4eb7-9b57-08d350bab832-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.692390 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.692179    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.692390 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.692272    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4eea30ae-084d-4eb7-9b57-08d350bab832-config-out\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.692390 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.692312    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.692390 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.692365    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.793376 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.793332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-config-volume\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.793568 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.793385    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4eea30ae-084d-4eb7-9b57-08d350bab832-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.793568 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.793433    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-web-config\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.793568 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.793467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eea30ae-084d-4eb7-9b57-08d350bab832-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.793568 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.793490    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84xng\" (UniqueName: \"kubernetes.io/projected/4eea30ae-084d-4eb7-9b57-08d350bab832-kube-api-access-84xng\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.793568 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.793533    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4eea30ae-084d-4eb7-9b57-08d350bab832-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.793568 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.793569    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.793880 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.793609    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4eea30ae-084d-4eb7-9b57-08d350bab832-config-out\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.793880 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.793632    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.793880 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.793670    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.793880 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.793713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.793880 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.793765    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.793880 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.793803    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eea30ae-084d-4eb7-9b57-08d350bab832-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.794266 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.794226    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4eea30ae-084d-4eb7-9b57-08d350bab832-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.794465 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.794436    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4eea30ae-084d-4eb7-9b57-08d350bab832-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.794706 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.794680    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4eea30ae-084d-4eb7-9b57-08d350bab832-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.796949 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.796915    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-config-volume\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.797695 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.797262    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4eea30ae-084d-4eb7-9b57-08d350bab832-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.797695 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.797535    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.797695 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.797576    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4eea30ae-084d-4eb7-9b57-08d350bab832-config-out\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.797695 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.797590    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.797937 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.797781    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.797937 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.797821    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.798354 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.798334    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.798475 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.798460    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4eea30ae-084d-4eb7-9b57-08d350bab832-web-config\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.801147 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.801123    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84xng\" (UniqueName: \"kubernetes.io/projected/4eea30ae-084d-4eb7-9b57-08d350bab832-kube-api-access-84xng\") pod \"alertmanager-main-0\" (UID: \"4eea30ae-084d-4eb7-9b57-08d350bab832\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:26.912716 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:26.912674    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 18:37:27.196467 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:27.196385    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 18:37:27.199664 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:37:27.199636    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eea30ae_084d_4eb7_9b57_08d350bab832.slice/crio-7802d069b755f4935bec0cfc95eaf215aa72959adbd973850cc05fd1e551934d WatchSource:0}: Error finding container 7802d069b755f4935bec0cfc95eaf215aa72959adbd973850cc05fd1e551934d: Status 404 returned error can't find the container with id 7802d069b755f4935bec0cfc95eaf215aa72959adbd973850cc05fd1e551934d
Sep 09 18:37:27.552405 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:27.552307    2575 generic.go:356] "Generic (PLEG): container finished" podID="4eea30ae-084d-4eb7-9b57-08d350bab832" containerID="a927f7b3e01f6aadaffb61defaeaa56ccdcbc390a7a46a28d47f6962db155f86" exitCode=0
Sep 09 18:37:27.552405 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:27.552390    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea30ae-084d-4eb7-9b57-08d350bab832","Type":"ContainerDied","Data":"a927f7b3e01f6aadaffb61defaeaa56ccdcbc390a7a46a28d47f6962db155f86"}
Sep 09 18:37:27.552936 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:27.552462    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea30ae-084d-4eb7-9b57-08d350bab832","Type":"ContainerStarted","Data":"7802d069b755f4935bec0cfc95eaf215aa72959adbd973850cc05fd1e551934d"}
Sep 09 18:37:28.323809 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.323772    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbf6cf70-9ee8-43e2-844d-b7ad093c2742" path="/var/lib/kubelet/pods/cbf6cf70-9ee8-43e2-844d-b7ad093c2742/volumes"
Sep 09 18:37:28.324382 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.324366    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5876879d86-94gxt"]
Sep 09 18:37:28.327411 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.327392    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.330077 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.330041    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 09 18:37:28.330221 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.330125    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-dmrzh\""
Sep 09 18:37:28.330221 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.330152    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 09 18:37:28.330340 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.330232    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 09 18:37:28.330537 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.330490    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 09 18:37:28.330677 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.330662    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 09 18:37:28.335402 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.335141    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 09 18:37:28.339882 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.339854    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5876879d86-94gxt"]
Sep 09 18:37:28.406429 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.406373    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef8d6649-c9cc-4933-b538-4041cd3f5bec-metrics-client-ca\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.406667 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.406452    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmvfb\" (UniqueName: \"kubernetes.io/projected/ef8d6649-c9cc-4933-b538-4041cd3f5bec-kube-api-access-jmvfb\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.406667 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.406491    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ef8d6649-c9cc-4933-b538-4041cd3f5bec-telemeter-client-tls\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.406667 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.406584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef8d6649-c9cc-4933-b538-4041cd3f5bec-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.406667 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.406657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef8d6649-c9cc-4933-b538-4041cd3f5bec-serving-certs-ca-bundle\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.406821 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.406687    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ef8d6649-c9cc-4933-b538-4041cd3f5bec-federate-client-tls\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.406821 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.406717    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef8d6649-c9cc-4933-b538-4041cd3f5bec-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.406821 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.406737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ef8d6649-c9cc-4933-b538-4041cd3f5bec-secret-telemeter-client\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.507784 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.507748    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ef8d6649-c9cc-4933-b538-4041cd3f5bec-telemeter-client-tls\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.507784 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.507790    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef8d6649-c9cc-4933-b538-4041cd3f5bec-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.508092 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.507825    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef8d6649-c9cc-4933-b538-4041cd3f5bec-serving-certs-ca-bundle\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.508092 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.507854    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ef8d6649-c9cc-4933-b538-4041cd3f5bec-federate-client-tls\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.508092 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.507874    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef8d6649-c9cc-4933-b538-4041cd3f5bec-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.508092 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.508022    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ef8d6649-c9cc-4933-b538-4041cd3f5bec-secret-telemeter-client\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.508092 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.508071    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef8d6649-c9cc-4933-b538-4041cd3f5bec-metrics-client-ca\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.508343 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.508136    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmvfb\" (UniqueName: \"kubernetes.io/projected/ef8d6649-c9cc-4933-b538-4041cd3f5bec-kube-api-access-jmvfb\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.508673 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.508646    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef8d6649-c9cc-4933-b538-4041cd3f5bec-serving-certs-ca-bundle\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.508857 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.508748    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef8d6649-c9cc-4933-b538-4041cd3f5bec-metrics-client-ca\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.508857 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.508759    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef8d6649-c9cc-4933-b538-4041cd3f5bec-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.510647 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.510618    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef8d6649-c9cc-4933-b538-4041cd3f5bec-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.511125 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.511101    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ef8d6649-c9cc-4933-b538-4041cd3f5bec-telemeter-client-tls\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.511176 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.511107    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ef8d6649-c9cc-4933-b538-4041cd3f5bec-secret-telemeter-client\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.511214 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.511188    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ef8d6649-c9cc-4933-b538-4041cd3f5bec-federate-client-tls\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.516351 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.516318    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmvfb\" (UniqueName: \"kubernetes.io/projected/ef8d6649-c9cc-4933-b538-4041cd3f5bec-kube-api-access-jmvfb\") pod \"telemeter-client-5876879d86-94gxt\" (UID: \"ef8d6649-c9cc-4933-b538-4041cd3f5bec\") " pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.561663 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.561620    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea30ae-084d-4eb7-9b57-08d350bab832","Type":"ContainerStarted","Data":"d703e88c99a807d05021bb0c1b4c47503838b4eec413522d7ee0ea210278736d"}
Sep 09 18:37:28.562050 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.561692    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea30ae-084d-4eb7-9b57-08d350bab832","Type":"ContainerStarted","Data":"04d73aac0cacef0fcae0789d3b4660310884632d5a55f2797abbbb0a5f95b088"}
Sep 09 18:37:28.562050 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.561728    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea30ae-084d-4eb7-9b57-08d350bab832","Type":"ContainerStarted","Data":"72a835c9db4a0de93d717c35f2f0fd13a7124091eab99cb4c7529f2e338d1bea"}
Sep 09 18:37:28.562050 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.561754    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea30ae-084d-4eb7-9b57-08d350bab832","Type":"ContainerStarted","Data":"06224bc224885b78bec3574f272f4abfbb5bcfa9a973b80c54dc8ee801daa83e"}
Sep 09 18:37:28.562050 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.561778    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea30ae-084d-4eb7-9b57-08d350bab832","Type":"ContainerStarted","Data":"71db142305be39aeeb4079c47d6292e83bedf495611353b764ad757919a46e43"}
Sep 09 18:37:28.562050 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.561812    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4eea30ae-084d-4eb7-9b57-08d350bab832","Type":"ContainerStarted","Data":"7ed7b410655c53473475302eefa9e8e32ae96317393899b13ff560e9e6b46b97"}
Sep 09 18:37:28.615347 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.615308    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 18:37:28.615887 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.615826    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="prometheus" containerID="cri-o://a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363" gracePeriod=600
Sep 09 18:37:28.616008 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.615892    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="kube-rbac-proxy" containerID="cri-o://fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0" gracePeriod=600
Sep 09 18:37:28.616066 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.616015    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="kube-rbac-proxy-web" containerID="cri-o://ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1" gracePeriod=600
Sep 09 18:37:28.616136 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.616059    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff" gracePeriod=600
Sep 09 18:37:28.616136 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.616114    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="thanos-sidecar" containerID="cri-o://01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23" gracePeriod=600
Sep 09 18:37:28.617130 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.616762    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="config-reloader" containerID="cri-o://5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea" gracePeriod=600
Sep 09 18:37:28.617371 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.617329    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.600699186 podStartE2EDuration="2.617312839s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.613653ms" isStatefulPod=false podCreationTimestamp="2026-09-09 18:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:37:28.616547459 +0000 UTC m=+336.853917274" watchObservedRunningTime="2026-09-09 18:37:28.617312839 +0000 UTC m=+336.854682653"
Sep 09 18:37:28.639742 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.639712    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5876879d86-94gxt"
Sep 09 18:37:28.856710 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.856675    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:28.911997 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.911895    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-2\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.911997 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.911942    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5009f37-b12d-4a23-af52-9372ca9b0ec4-config-out\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.911997 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.911968    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-db\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.912276 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912000    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-serving-certs-ca-bundle\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.912276 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912033    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdqz2\" (UniqueName: \"kubernetes.io/projected/b5009f37-b12d-4a23-af52-9372ca9b0ec4-kube-api-access-sdqz2\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.912276 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912086    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-web-config\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.912276 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912113    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-metrics-client-certs\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.912481 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912441    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912681    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-config\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912713    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912731    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-thanos-prometheus-http-client-file\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912767    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-grpc-tls\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912793    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-1\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912840    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-metrics-client-ca\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912864    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5009f37-b12d-4a23-af52-9372ca9b0ec4-tls-assets\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912893    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-trusted-ca-bundle\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912920    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.912955    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.913010    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-kubelet-serving-ca-bundle\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.913039    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-tls\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.913077    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-kube-rbac-proxy\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.913141    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-0\") pod \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\" (UID: \"b5009f37-b12d-4a23-af52-9372ca9b0ec4\") "
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.913442    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:28.913752 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.913462    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:28.914692 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.914310    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:37:28.914692 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.914414    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:37:28.914825 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.914723    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:37:28.915400 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.915245    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5009f37-b12d-4a23-af52-9372ca9b0ec4-config-out" (OuterVolumeSpecName: "config-out") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:37:28.915646 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.915610    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:37:28.916051 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.915999    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:37:28.916051 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.916008    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:37:28.916327 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.916292    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:28.917293 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.917253    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:28.917878 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.917838    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-config" (OuterVolumeSpecName: "config") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:28.917878 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.917824    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:28.918198 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.918165    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5009f37-b12d-4a23-af52-9372ca9b0ec4-kube-api-access-sdqz2" (OuterVolumeSpecName: "kube-api-access-sdqz2") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "kube-api-access-sdqz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:37:28.918272 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.918244    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:28.918603 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.918567    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5009f37-b12d-4a23-af52-9372ca9b0ec4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:37:28.918699 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.918656    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:28.919248 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.919221    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:28.919400 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.919379    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:28.923660 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.923626    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5876879d86-94gxt"]
Sep 09 18:37:28.927939 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:37:28.927906    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8d6649_c9cc_4933_b538_4041cd3f5bec.slice/crio-2e66da480b05cc4f0b04649751ee34c12d8a60210d3c8f8dbf6ecd9bb86612a3 WatchSource:0}: Error finding container 2e66da480b05cc4f0b04649751ee34c12d8a60210d3c8f8dbf6ecd9bb86612a3: Status 404 returned error can't find the container with id 2e66da480b05cc4f0b04649751ee34c12d8a60210d3c8f8dbf6ecd9bb86612a3
Sep 09 18:37:28.929366 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.929333    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-web-config" (OuterVolumeSpecName: "web-config") pod "b5009f37-b12d-4a23-af52-9372ca9b0ec4" (UID: "b5009f37-b12d-4a23-af52-9372ca9b0ec4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:37:28.930056 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:28.930038    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 18:37:29.014683 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014631    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-db\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014683 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014670    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdqz2\" (UniqueName: \"kubernetes.io/projected/b5009f37-b12d-4a23-af52-9372ca9b0ec4-kube-api-access-sdqz2\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014683 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014682    2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-web-config\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014683 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014692    2575 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-metrics-client-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014701    2575 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-config\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014711    2575 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014720    2575 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-grpc-tls\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014728    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014736    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-metrics-client-ca\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014745    2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5009f37-b12d-4a23-af52-9372ca9b0ec4-tls-assets\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014754    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014763    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014773    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014781    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014790    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014797    2575 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5009f37-b12d-4a23-af52-9372ca9b0ec4-secret-kube-rbac-proxy\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014806    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b5009f37-b12d-4a23-af52-9372ca9b0ec4-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.014945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.014814    2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5009f37-b12d-4a23-af52-9372ca9b0ec4-config-out\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:37:29.567965 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.567926    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5876879d86-94gxt" event={"ID":"ef8d6649-c9cc-4933-b538-4041cd3f5bec","Type":"ContainerStarted","Data":"2e66da480b05cc4f0b04649751ee34c12d8a60210d3c8f8dbf6ecd9bb86612a3"}
Sep 09 18:37:29.571272 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571237    2575 generic.go:356] "Generic (PLEG): container finished" podID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerID="9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff" exitCode=0
Sep 09 18:37:29.571272 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571268    2575 generic.go:356] "Generic (PLEG): container finished" podID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerID="fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0" exitCode=0
Sep 09 18:37:29.571272 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571278    2575 generic.go:356] "Generic (PLEG): container finished" podID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerID="ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1" exitCode=0
Sep 09 18:37:29.571517 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571288    2575 generic.go:356] "Generic (PLEG): container finished" podID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerID="01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23" exitCode=0
Sep 09 18:37:29.571517 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571299    2575 generic.go:356] "Generic (PLEG): container finished" podID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerID="5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea" exitCode=0
Sep 09 18:37:29.571517 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571307    2575 generic.go:356] "Generic (PLEG): container finished" podID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerID="a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363" exitCode=0
Sep 09 18:37:29.571517 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571310    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerDied","Data":"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"}
Sep 09 18:37:29.571517 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571331    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.571517 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571371    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerDied","Data":"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"}
Sep 09 18:37:29.571517 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571399    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerDied","Data":"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"}
Sep 09 18:37:29.571517 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571424    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerDied","Data":"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"}
Sep 09 18:37:29.571517 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571449    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerDied","Data":"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"}
Sep 09 18:37:29.571517 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571457    2575 scope.go:117] "RemoveContainer" containerID="9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"
Sep 09 18:37:29.571517 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571474    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerDied","Data":"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"}
Sep 09 18:37:29.571517 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.571511    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b5009f37-b12d-4a23-af52-9372ca9b0ec4","Type":"ContainerDied","Data":"384e0db2fad6bcf6e71d5fa12c375be5fce5cf5d6556cf40dc2d619d194cfc21"}
Sep 09 18:37:29.580216 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.580190    2575 scope.go:117] "RemoveContainer" containerID="fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"
Sep 09 18:37:29.588369 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.588345    2575 scope.go:117] "RemoveContainer" containerID="ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"
Sep 09 18:37:29.596360 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.596326    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 18:37:29.597375 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.597353    2575 scope.go:117] "RemoveContainer" containerID="01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"
Sep 09 18:37:29.600802 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.600778    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 18:37:29.606101 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.606078    2575 scope.go:117] "RemoveContainer" containerID="5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"
Sep 09 18:37:29.614203 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.614181    2575 scope.go:117] "RemoveContainer" containerID="a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"
Sep 09 18:37:29.624184 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624109    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 18:37:29.624602 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624484    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="prometheus"
Sep 09 18:37:29.624602 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624521    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="prometheus"
Sep 09 18:37:29.624602 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624536    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="config-reloader"
Sep 09 18:37:29.624602 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624543    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="config-reloader"
Sep 09 18:37:29.624602 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624568    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="kube-rbac-proxy-web"
Sep 09 18:37:29.624602 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624576    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="kube-rbac-proxy-web"
Sep 09 18:37:29.624602 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624583    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="kube-rbac-proxy-thanos"
Sep 09 18:37:29.624602 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624591    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="kube-rbac-proxy-thanos"
Sep 09 18:37:29.625014 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624608    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="init-config-reloader"
Sep 09 18:37:29.625014 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624616    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="init-config-reloader"
Sep 09 18:37:29.625014 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624626    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="kube-rbac-proxy"
Sep 09 18:37:29.625014 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624633    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="kube-rbac-proxy"
Sep 09 18:37:29.625014 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624647    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="thanos-sidecar"
Sep 09 18:37:29.625014 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624654    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="thanos-sidecar"
Sep 09 18:37:29.625014 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624737    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="prometheus"
Sep 09 18:37:29.625014 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624749    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="kube-rbac-proxy-web"
Sep 09 18:37:29.625014 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624759    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="kube-rbac-proxy-thanos"
Sep 09 18:37:29.625014 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624769    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="thanos-sidecar"
Sep 09 18:37:29.625014 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624777    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="kube-rbac-proxy"
Sep 09 18:37:29.625014 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.624787    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" containerName="config-reloader"
Sep 09 18:37:29.625922 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.625754    2575 scope.go:117] "RemoveContainer" containerID="63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"
Sep 09 18:37:29.629243 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.629219    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.631705 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.631671    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 09 18:37:29.632027 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.631999    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 09 18:37:29.632243 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.632221    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 09 18:37:29.632345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.632272    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-pbhr060igvkh\""
Sep 09 18:37:29.632345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.632023    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-flv6g\""
Sep 09 18:37:29.632458 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.632025    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 09 18:37:29.632458 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.632032    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 09 18:37:29.632458 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.632396    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 09 18:37:29.632458 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.632133    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 09 18:37:29.632458 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.632138    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 09 18:37:29.632768 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.632174    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 09 18:37:29.632768 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.632023    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 09 18:37:29.632768 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.632684    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 09 18:37:29.632768 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.632754    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 09 18:37:29.634685 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.634659    2575 scope.go:117] "RemoveContainer" containerID="9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"
Sep 09 18:37:29.635097 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:37:29.635068    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff\": container with ID starting with 9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff not found: ID does not exist" containerID="9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"
Sep 09 18:37:29.635185 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.635110    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"} err="failed to get container status \"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff\": rpc error: code = NotFound desc = could not find container \"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff\": container with ID starting with 9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff not found: ID does not exist"
Sep 09 18:37:29.635185 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.635135    2575 scope.go:117] "RemoveContainer" containerID="fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"
Sep 09 18:37:29.635421 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.635405    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 09 18:37:29.635534 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:37:29.635458    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0\": container with ID starting with fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0 not found: ID does not exist" containerID="fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"
Sep 09 18:37:29.635534 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.635482    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"} err="failed to get container status \"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0\": rpc error: code = NotFound desc = could not find container \"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0\": container with ID starting with fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0 not found: ID does not exist"
Sep 09 18:37:29.635534 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.635523    2575 scope.go:117] "RemoveContainer" containerID="ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"
Sep 09 18:37:29.635888 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:37:29.635859    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1\": container with ID starting with ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1 not found: ID does not exist" containerID="ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"
Sep 09 18:37:29.635970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.635892    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"} err="failed to get container status \"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1\": rpc error: code = NotFound desc = could not find container \"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1\": container with ID starting with ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1 not found: ID does not exist"
Sep 09 18:37:29.635970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.635907    2575 scope.go:117] "RemoveContainer" containerID="01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"
Sep 09 18:37:29.636200 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:37:29.636183    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23\": container with ID starting with 01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23 not found: ID does not exist" containerID="01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"
Sep 09 18:37:29.636247 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.636204    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"} err="failed to get container status \"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23\": rpc error: code = NotFound desc = could not find container \"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23\": container with ID starting with 01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23 not found: ID does not exist"
Sep 09 18:37:29.636247 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.636218    2575 scope.go:117] "RemoveContainer" containerID="5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"
Sep 09 18:37:29.636538 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:37:29.636519    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea\": container with ID starting with 5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea not found: ID does not exist" containerID="5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"
Sep 09 18:37:29.636613 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.636541    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"} err="failed to get container status \"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea\": rpc error: code = NotFound desc = could not find container \"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea\": container with ID starting with 5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea not found: ID does not exist"
Sep 09 18:37:29.636613 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.636555    2575 scope.go:117] "RemoveContainer" containerID="a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"
Sep 09 18:37:29.636845 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:37:29.636820    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363\": container with ID starting with a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363 not found: ID does not exist" containerID="a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"
Sep 09 18:37:29.636940 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.636850    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"} err="failed to get container status \"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363\": rpc error: code = NotFound desc = could not find container \"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363\": container with ID starting with a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363 not found: ID does not exist"
Sep 09 18:37:29.636940 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.636866    2575 scope.go:117] "RemoveContainer" containerID="63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"
Sep 09 18:37:29.637178 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:37:29.637161    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77\": container with ID starting with 63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77 not found: ID does not exist" containerID="63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"
Sep 09 18:37:29.637266 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.637182    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"} err="failed to get container status \"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77\": rpc error: code = NotFound desc = could not find container \"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77\": container with ID starting with 63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77 not found: ID does not exist"
Sep 09 18:37:29.637266 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.637196    2575 scope.go:117] "RemoveContainer" containerID="9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"
Sep 09 18:37:29.637472 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.637451    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"} err="failed to get container status \"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff\": rpc error: code = NotFound desc = could not find container \"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff\": container with ID starting with 9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff not found: ID does not exist"
Sep 09 18:37:29.637600 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.637474    2575 scope.go:117] "RemoveContainer" containerID="fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"
Sep 09 18:37:29.637785 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.637746    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"} err="failed to get container status \"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0\": rpc error: code = NotFound desc = could not find container \"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0\": container with ID starting with fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0 not found: ID does not exist"
Sep 09 18:37:29.637785 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.637773    2575 scope.go:117] "RemoveContainer" containerID="ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"
Sep 09 18:37:29.637958 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.637936    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 09 18:37:29.638064 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.638034    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"} err="failed to get container status \"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1\": rpc error: code = NotFound desc = could not find container \"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1\": container with ID starting with ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1 not found: ID does not exist"
Sep 09 18:37:29.638064 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.638055    2575 scope.go:117] "RemoveContainer" containerID="01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"
Sep 09 18:37:29.638542 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.638346    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"} err="failed to get container status \"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23\": rpc error: code = NotFound desc = could not find container \"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23\": container with ID starting with 01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23 not found: ID does not exist"
Sep 09 18:37:29.638542 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.638371    2575 scope.go:117] "RemoveContainer" containerID="5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"
Sep 09 18:37:29.638748 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.638725    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"} err="failed to get container status \"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea\": rpc error: code = NotFound desc = could not find container \"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea\": container with ID starting with 5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea not found: ID does not exist"
Sep 09 18:37:29.638790 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.638749    2575 scope.go:117] "RemoveContainer" containerID="a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"
Sep 09 18:37:29.639015 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.638991    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"} err="failed to get container status \"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363\": rpc error: code = NotFound desc = could not find container \"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363\": container with ID starting with a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363 not found: ID does not exist"
Sep 09 18:37:29.639114 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.639016    2575 scope.go:117] "RemoveContainer" containerID="63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"
Sep 09 18:37:29.639940 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.639674    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"} err="failed to get container status \"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77\": rpc error: code = NotFound desc = could not find container \"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77\": container with ID starting with 63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77 not found: ID does not exist"
Sep 09 18:37:29.639940 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.639722    2575 scope.go:117] "RemoveContainer" containerID="9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"
Sep 09 18:37:29.640345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.640107    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"} err="failed to get container status \"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff\": rpc error: code = NotFound desc = could not find container \"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff\": container with ID starting with 9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff not found: ID does not exist"
Sep 09 18:37:29.640345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.640150    2575 scope.go:117] "RemoveContainer" containerID="fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"
Sep 09 18:37:29.641142 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.641109    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"} err="failed to get container status \"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0\": rpc error: code = NotFound desc = could not find container \"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0\": container with ID starting with fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0 not found: ID does not exist"
Sep 09 18:37:29.641250 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.641145    2575 scope.go:117] "RemoveContainer" containerID="ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"
Sep 09 18:37:29.642005 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.641929    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"} err="failed to get container status \"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1\": rpc error: code = NotFound desc = could not find container \"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1\": container with ID starting with ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1 not found: ID does not exist"
Sep 09 18:37:29.642005 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.641958    2575 scope.go:117] "RemoveContainer" containerID="01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"
Sep 09 18:37:29.642345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.642324    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 18:37:29.642783 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.642756    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"} err="failed to get container status \"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23\": rpc error: code = NotFound desc = could not find container \"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23\": container with ID starting with 01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23 not found: ID does not exist"
Sep 09 18:37:29.642888 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.642786    2575 scope.go:117] "RemoveContainer" containerID="5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"
Sep 09 18:37:29.643084 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.643056    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"} err="failed to get container status \"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea\": rpc error: code = NotFound desc = could not find container \"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea\": container with ID starting with 5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea not found: ID does not exist"
Sep 09 18:37:29.643084 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.643083    2575 scope.go:117] "RemoveContainer" containerID="a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"
Sep 09 18:37:29.643435 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.643392    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"} err="failed to get container status \"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363\": rpc error: code = NotFound desc = could not find container \"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363\": container with ID starting with a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363 not found: ID does not exist"
Sep 09 18:37:29.643546 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.643436    2575 scope.go:117] "RemoveContainer" containerID="63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"
Sep 09 18:37:29.643990 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.643863    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"} err="failed to get container status \"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77\": rpc error: code = NotFound desc = could not find container \"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77\": container with ID starting with 63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77 not found: ID does not exist"
Sep 09 18:37:29.643990 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.643904    2575 scope.go:117] "RemoveContainer" containerID="9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"
Sep 09 18:37:29.644301 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.644205    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"} err="failed to get container status \"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff\": rpc error: code = NotFound desc = could not find container \"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff\": container with ID starting with 9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff not found: ID does not exist"
Sep 09 18:37:29.644301 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.644229    2575 scope.go:117] "RemoveContainer" containerID="fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"
Sep 09 18:37:29.644614 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.644526    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"} err="failed to get container status \"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0\": rpc error: code = NotFound desc = could not find container \"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0\": container with ID starting with fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0 not found: ID does not exist"
Sep 09 18:37:29.644614 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.644549    2575 scope.go:117] "RemoveContainer" containerID="ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"
Sep 09 18:37:29.644907 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.644881    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"} err="failed to get container status \"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1\": rpc error: code = NotFound desc = could not find container \"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1\": container with ID starting with ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1 not found: ID does not exist"
Sep 09 18:37:29.644994 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.644908    2575 scope.go:117] "RemoveContainer" containerID="01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"
Sep 09 18:37:29.645168 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.645150    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"} err="failed to get container status \"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23\": rpc error: code = NotFound desc = could not find container \"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23\": container with ID starting with 01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23 not found: ID does not exist"
Sep 09 18:37:29.645239 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.645168    2575 scope.go:117] "RemoveContainer" containerID="5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"
Sep 09 18:37:29.645553 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.645456    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"} err="failed to get container status \"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea\": rpc error: code = NotFound desc = could not find container \"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea\": container with ID starting with 5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea not found: ID does not exist"
Sep 09 18:37:29.645553 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.645480    2575 scope.go:117] "RemoveContainer" containerID="a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"
Sep 09 18:37:29.646081 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.645986    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"} err="failed to get container status \"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363\": rpc error: code = NotFound desc = could not find container \"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363\": container with ID starting with a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363 not found: ID does not exist"
Sep 09 18:37:29.646081 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.646013    2575 scope.go:117] "RemoveContainer" containerID="63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"
Sep 09 18:37:29.646672 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.646644    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"} err="failed to get container status \"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77\": rpc error: code = NotFound desc = could not find container \"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77\": container with ID starting with 63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77 not found: ID does not exist"
Sep 09 18:37:29.646672 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.646669    2575 scope.go:117] "RemoveContainer" containerID="9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"
Sep 09 18:37:29.647000 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.646978    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"} err="failed to get container status \"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff\": rpc error: code = NotFound desc = could not find container \"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff\": container with ID starting with 9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff not found: ID does not exist"
Sep 09 18:37:29.647055 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.647000    2575 scope.go:117] "RemoveContainer" containerID="fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"
Sep 09 18:37:29.647255 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.647234    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"} err="failed to get container status \"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0\": rpc error: code = NotFound desc = could not find container \"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0\": container with ID starting with fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0 not found: ID does not exist"
Sep 09 18:37:29.647325 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.647255    2575 scope.go:117] "RemoveContainer" containerID="ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"
Sep 09 18:37:29.647570 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.647542    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"} err="failed to get container status \"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1\": rpc error: code = NotFound desc = could not find container \"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1\": container with ID starting with ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1 not found: ID does not exist"
Sep 09 18:37:29.647651 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.647574    2575 scope.go:117] "RemoveContainer" containerID="01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"
Sep 09 18:37:29.647903 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.647879    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"} err="failed to get container status \"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23\": rpc error: code = NotFound desc = could not find container \"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23\": container with ID starting with 01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23 not found: ID does not exist"
Sep 09 18:37:29.647980 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.647904    2575 scope.go:117] "RemoveContainer" containerID="5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"
Sep 09 18:37:29.648189 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.648150    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"} err="failed to get container status \"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea\": rpc error: code = NotFound desc = could not find container \"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea\": container with ID starting with 5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea not found: ID does not exist"
Sep 09 18:37:29.648189 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.648177    2575 scope.go:117] "RemoveContainer" containerID="a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"
Sep 09 18:37:29.648485 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.648464    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"} err="failed to get container status \"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363\": rpc error: code = NotFound desc = could not find container \"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363\": container with ID starting with a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363 not found: ID does not exist"
Sep 09 18:37:29.648616 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.648484    2575 scope.go:117] "RemoveContainer" containerID="63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"
Sep 09 18:37:29.648838 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.648811    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"} err="failed to get container status \"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77\": rpc error: code = NotFound desc = could not find container \"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77\": container with ID starting with 63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77 not found: ID does not exist"
Sep 09 18:37:29.648930 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.648839    2575 scope.go:117] "RemoveContainer" containerID="9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"
Sep 09 18:37:29.649110 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.649090    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff"} err="failed to get container status \"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff\": rpc error: code = NotFound desc = could not find container \"9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff\": container with ID starting with 9d2ecf56189ea05ff3ef30f9a9bbf9a4a03992054f98d251ca7c6f99760f38ff not found: ID does not exist"
Sep 09 18:37:29.649190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.649109    2575 scope.go:117] "RemoveContainer" containerID="fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"
Sep 09 18:37:29.649394 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.649368    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0"} err="failed to get container status \"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0\": rpc error: code = NotFound desc = could not find container \"fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0\": container with ID starting with fac19e43e596d03eb9ee08d0a9e129f7744f312eb95dd4aaa29345a6e4a478c0 not found: ID does not exist"
Sep 09 18:37:29.649394 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.649394    2575 scope.go:117] "RemoveContainer" containerID="ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"
Sep 09 18:37:29.649822 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.649787    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1"} err="failed to get container status \"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1\": rpc error: code = NotFound desc = could not find container \"ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1\": container with ID starting with ccb1e30714ec351d0dc09aa73ec886c9abf5fc19262bf1e058b60c8ec140e1a1 not found: ID does not exist"
Sep 09 18:37:29.649937 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.649824    2575 scope.go:117] "RemoveContainer" containerID="01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"
Sep 09 18:37:29.650178 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.650151    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23"} err="failed to get container status \"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23\": rpc error: code = NotFound desc = could not find container \"01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23\": container with ID starting with 01b8174a64d5c6eb114cb36272241a467d747b503708c0592c97abc520b8cc23 not found: ID does not exist"
Sep 09 18:37:29.650178 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.650177    2575 scope.go:117] "RemoveContainer" containerID="5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"
Sep 09 18:37:29.650489 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.650462    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea"} err="failed to get container status \"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea\": rpc error: code = NotFound desc = could not find container \"5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea\": container with ID starting with 5900bb0f233b9f4301accb0fda26c90693fada2409ab3d628e961f44613154ea not found: ID does not exist"
Sep 09 18:37:29.650601 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.650490    2575 scope.go:117] "RemoveContainer" containerID="a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"
Sep 09 18:37:29.650829 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.650801    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363"} err="failed to get container status \"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363\": rpc error: code = NotFound desc = could not find container \"a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363\": container with ID starting with a59aad16e5dafdeaf2c482adefa9397951dc7cefbeef99c5c64ff41d7cfab363 not found: ID does not exist"
Sep 09 18:37:29.650917 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.650831    2575 scope.go:117] "RemoveContainer" containerID="63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"
Sep 09 18:37:29.651120 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.651100    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77"} err="failed to get container status \"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77\": rpc error: code = NotFound desc = could not find container \"63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77\": container with ID starting with 63179395afa632961a3ed6abe4a38e4b47f694431372515792a2cec29e515b77 not found: ID does not exist"
Sep 09 18:37:29.720032 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.719987    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720032 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720035    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720281 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720053    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720281 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720090    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720281 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720127    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720281 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720162    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720281 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720219    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720281 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720281 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720265    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-config\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720663 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720288    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b492f3c-9415-4ea1-854c-ca896f244e49-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720663 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720305    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720663 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720334    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720663 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720374    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720663 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720398    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720663 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720426    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwpgz\" (UniqueName: \"kubernetes.io/projected/5b492f3c-9415-4ea1-854c-ca896f244e49-kube-api-access-wwpgz\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720663 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720490    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720663 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720542    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b492f3c-9415-4ea1-854c-ca896f244e49-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720663 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720663 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.720663 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.720643    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821346 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821346 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821342    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821619 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821361    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwpgz\" (UniqueName: \"kubernetes.io/projected/5b492f3c-9415-4ea1-854c-ca896f244e49-kube-api-access-wwpgz\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821619 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821619 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821401    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b492f3c-9415-4ea1-854c-ca896f244e49-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821619 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821544    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821619 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821583    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821619 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821619    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821913 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821675    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821913 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821707    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821913 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821732    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821913 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821754    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821913 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821789    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821913 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821811    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821913 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821876    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.821913 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821897    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.822308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821929    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-config\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.822308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821954    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b492f3c-9415-4ea1-854c-ca896f244e49-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.822308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.821975    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.822308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.822012    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.822308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.822076    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.822308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.822166    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.822613 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.822415    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.822613 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.822416    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.822763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.822737    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.824669 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.824639    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b492f3c-9415-4ea1-854c-ca896f244e49-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.825195 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.824856    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.825195 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.824879    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.825353 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.825233    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-config\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.825868 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.825547    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.825868 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.825710    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.826041 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.825977    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.826377 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.826349    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.826476 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.826458    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b492f3c-9415-4ea1-854c-ca896f244e49-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.826980 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.826953    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.827077 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.827037    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b492f3c-9415-4ea1-854c-ca896f244e49-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.827546 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.827526    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.827930 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.827905    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.828120 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.828100    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b492f3c-9415-4ea1-854c-ca896f244e49-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.829332 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.829311    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwpgz\" (UniqueName: \"kubernetes.io/projected/5b492f3c-9415-4ea1-854c-ca896f244e49-kube-api-access-wwpgz\") pod \"prometheus-k8s-0\" (UID: \"5b492f3c-9415-4ea1-854c-ca896f244e49\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:29.943125 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:29.943030    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:37:30.224240 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:30.224213    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 09 18:37:30.226220 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:37:30.226186    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b492f3c_9415_4ea1_854c_ca896f244e49.slice/crio-3507e8ebfe99831c59b27491738fddb57b8433bf40121604fdc2dc0ffea63f9a WatchSource:0}: Error finding container 3507e8ebfe99831c59b27491738fddb57b8433bf40121604fdc2dc0ffea63f9a: Status 404 returned error can't find the container with id 3507e8ebfe99831c59b27491738fddb57b8433bf40121604fdc2dc0ffea63f9a
Sep 09 18:37:30.323538 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:30.322083    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5009f37-b12d-4a23-af52-9372ca9b0ec4" path="/var/lib/kubelet/pods/b5009f37-b12d-4a23-af52-9372ca9b0ec4/volumes"
Sep 09 18:37:30.579138 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:30.579043    2575 generic.go:356] "Generic (PLEG): container finished" podID="5b492f3c-9415-4ea1-854c-ca896f244e49" containerID="961c51aab1024a7f2de4646a33a630ec3bf8f6c64b66388fd6dec781d897bfe6" exitCode=0
Sep 09 18:37:30.579604 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:30.579172    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b492f3c-9415-4ea1-854c-ca896f244e49","Type":"ContainerDied","Data":"961c51aab1024a7f2de4646a33a630ec3bf8f6c64b66388fd6dec781d897bfe6"}
Sep 09 18:37:30.579604 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:30.579242    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b492f3c-9415-4ea1-854c-ca896f244e49","Type":"ContainerStarted","Data":"3507e8ebfe99831c59b27491738fddb57b8433bf40121604fdc2dc0ffea63f9a"}
Sep 09 18:37:31.589570 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:31.589461    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b492f3c-9415-4ea1-854c-ca896f244e49","Type":"ContainerStarted","Data":"971e914e173e2b88b06437be158a91e65c962fac2388caa45b952b57bef3acda"}
Sep 09 18:37:31.589570 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:31.589565    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b492f3c-9415-4ea1-854c-ca896f244e49","Type":"ContainerStarted","Data":"570da5487b755573e758289c545f28acdc07f33b30a1b7577fa0ca14227be04f"}
Sep 09 18:37:31.590047 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:31.589608    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b492f3c-9415-4ea1-854c-ca896f244e49","Type":"ContainerStarted","Data":"8b094cfceead51788bce0cb0b9c3f684bf507ed1a813449154ea036d1e399714"}
Sep 09 18:37:31.590047 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:31.589645    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b492f3c-9415-4ea1-854c-ca896f244e49","Type":"ContainerStarted","Data":"190f6cf3261997743603a681651244a9b3d57be92bb4671ed44924804a442d8c"}
Sep 09 18:37:31.590047 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:31.589681    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b492f3c-9415-4ea1-854c-ca896f244e49","Type":"ContainerStarted","Data":"e8341775fc791db95195104ddab1ce7fe6473ce77a894fba1e77e9018b9b599a"}
Sep 09 18:37:31.590047 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:31.589716    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b492f3c-9415-4ea1-854c-ca896f244e49","Type":"ContainerStarted","Data":"78521c4d730f7a0ce2f0b2874575c2fadfeff9e0c6ab99e7834de28865463b19"}
Sep 09 18:37:31.591882 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:31.591852    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5876879d86-94gxt" event={"ID":"ef8d6649-c9cc-4933-b538-4041cd3f5bec","Type":"ContainerStarted","Data":"c511363aead7def6e4eb19a1bdbbd808700defa8cccf6fc2773131aabc1675b3"}
Sep 09 18:37:31.591987 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:31.591909    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5876879d86-94gxt" event={"ID":"ef8d6649-c9cc-4933-b538-4041cd3f5bec","Type":"ContainerStarted","Data":"98d51439f036e4d32d4b808ba8c6cae4155f221e9a09cf9d5209ad9dde9423e8"}
Sep 09 18:37:31.591987 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:31.591939    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5876879d86-94gxt" event={"ID":"ef8d6649-c9cc-4933-b538-4041cd3f5bec","Type":"ContainerStarted","Data":"43e3d764854302950efdf994e334e17f8ba1e91902a4bb01b1304138b7a1e231"}
Sep 09 18:37:31.616583 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:31.614360    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.58913079 podStartE2EDuration="2.614338151s" totalImagesPullingTime="0s" totalInitContainerRuntime="25.207361ms" isStatefulPod=false podCreationTimestamp="2026-09-09 18:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:37:31.613343583 +0000 UTC m=+339.850713411" watchObservedRunningTime="2026-09-09 18:37:31.614338151 +0000 UTC m=+339.851707968"
Sep 09 18:37:31.633305 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:31.633252    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5876879d86-94gxt" podStartSLOduration=1.550228681 podStartE2EDuration="3.633208654s" totalImagesPullingTime="2.082979973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:37:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:37:31.632198036 +0000 UTC m=+339.869567876" watchObservedRunningTime="2026-09-09 18:37:31.633208654 +0000 UTC m=+339.870578470"
Sep 09 18:37:34.943396 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:37:34.943355    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:38:29.944163 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:38:29.944112    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:38:29.960554 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:38:29.960520    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:38:30.859606 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:38:30.859574    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 09 18:41:10.256657 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.256576    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"]
Sep 09 18:41:10.259229 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.259208    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.262700 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.262676    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 18:41:10.262909 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.262891    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jqmk5\""
Sep 09 18:41:10.262992 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.262949    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 18:41:10.263048 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.262987    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 18:41:10.263048 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.262989    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 18:41:10.309586 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.309553    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"]
Sep 09 18:41:10.387213 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.387170    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1fd60a9f-60c0-4b72-800f-e117c90283c8-manifests\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.387432 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.387266    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fd60a9f-60c0-4b72-800f-e117c90283c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.387432 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.387332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1fd60a9f-60c0-4b72-800f-e117c90283c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.387432 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.387357    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fd60a9f-60c0-4b72-800f-e117c90283c8-tmp\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.387432 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.387386    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bwf6\" (UniqueName: \"kubernetes.io/projected/1fd60a9f-60c0-4b72-800f-e117c90283c8-kube-api-access-7bwf6\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.488866 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.488819    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1fd60a9f-60c0-4b72-800f-e117c90283c8-manifests\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.488866 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.488868    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fd60a9f-60c0-4b72-800f-e117c90283c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.489113 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.488926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1fd60a9f-60c0-4b72-800f-e117c90283c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.489113 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.488951    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fd60a9f-60c0-4b72-800f-e117c90283c8-tmp\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.489113 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.488978    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bwf6\" (UniqueName: \"kubernetes.io/projected/1fd60a9f-60c0-4b72-800f-e117c90283c8-kube-api-access-7bwf6\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.489259 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.489240    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1fd60a9f-60c0-4b72-800f-e117c90283c8-manifests\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.491201 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.491169    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fd60a9f-60c0-4b72-800f-e117c90283c8-tmp\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.491448 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.491428    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fd60a9f-60c0-4b72-800f-e117c90283c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.491529 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.491453    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1fd60a9f-60c0-4b72-800f-e117c90283c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.510928 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.510855    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bwf6\" (UniqueName: \"kubernetes.io/projected/1fd60a9f-60c0-4b72-800f-e117c90283c8-kube-api-access-7bwf6\") pod \"opendatahub-operator-controller-manager-5f66f47469-nnn72\" (UID: \"1fd60a9f-60c0-4b72-800f-e117c90283c8\") " pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.570541 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.570482    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:10.872086 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:10.872049    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"]
Sep 09 18:41:10.875837 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:41:10.875795    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd60a9f_60c0_4b72_800f_e117c90283c8.slice/crio-4174405da3314b451305ee0a541337310d262d166e8ce755de84a77c9bb3ca8a WatchSource:0}: Error finding container 4174405da3314b451305ee0a541337310d262d166e8ce755de84a77c9bb3ca8a: Status 404 returned error can't find the container with id 4174405da3314b451305ee0a541337310d262d166e8ce755de84a77c9bb3ca8a
Sep 09 18:41:11.509282 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:11.509238    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72" event={"ID":"1fd60a9f-60c0-4b72-800f-e117c90283c8","Type":"ContainerStarted","Data":"4174405da3314b451305ee0a541337310d262d166e8ce755de84a77c9bb3ca8a"}
Sep 09 18:41:14.528469 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:14.528367    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72" event={"ID":"1fd60a9f-60c0-4b72-800f-e117c90283c8","Type":"ContainerStarted","Data":"e350f4dd07e203e1c885153608942e8392207458cd226d8a53816a8c32e62302"}
Sep 09 18:41:15.535278 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:15.535240    2575 generic.go:356] "Generic (PLEG): container finished" podID="1fd60a9f-60c0-4b72-800f-e117c90283c8" containerID="e350f4dd07e203e1c885153608942e8392207458cd226d8a53816a8c32e62302" exitCode=0
Sep 09 18:41:15.535741 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:15.535340    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72" event={"ID":"1fd60a9f-60c0-4b72-800f-e117c90283c8","Type":"ContainerDied","Data":"e350f4dd07e203e1c885153608942e8392207458cd226d8a53816a8c32e62302"}
Sep 09 18:41:16.542283 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:16.542246    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72" event={"ID":"1fd60a9f-60c0-4b72-800f-e117c90283c8","Type":"ContainerStarted","Data":"92dde12666a63741d258831107225161149043ea0052282b59353cf4878e7a9d"}
Sep 09 18:41:16.542831 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:16.542356    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:16.575418 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:16.575355    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72" podStartSLOduration=1.815456936 podStartE2EDuration="6.575335754s" totalImagesPullingTime="3.621274699s" totalInitContainerRuntime="1.138604119s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:41:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:41:16.574434807 +0000 UTC m=+564.811804624" watchObservedRunningTime="2026-09-09 18:41:16.575335754 +0000 UTC m=+564.812705569"
Sep 09 18:41:27.550021 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:27.549982    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f66f47469-nnn72"
Sep 09 18:41:32.412492 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.412399    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx"]
Sep 09 18:41:32.423757 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.423730    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx"
Sep 09 18:41:32.429539 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.428655    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 18:41:32.429539 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.428726    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 18:41:32.429539 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.429015    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 18:41:32.429539 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.429305    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 18:41:32.454719 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.454684    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx"]
Sep 09 18:41:32.495944 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.495900    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6f112a-49fb-4edf-af39-8603398fab38-tls-certs\") pod \"kube-auth-proxy-799dfc8bc6-flvvx\" (UID: \"3f6f112a-49fb-4edf-af39-8603398fab38\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx"
Sep 09 18:41:32.496143 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.495961    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xjsf\" (UniqueName: \"kubernetes.io/projected/3f6f112a-49fb-4edf-af39-8603398fab38-kube-api-access-9xjsf\") pod \"kube-auth-proxy-799dfc8bc6-flvvx\" (UID: \"3f6f112a-49fb-4edf-af39-8603398fab38\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx"
Sep 09 18:41:32.496143 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.496017    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f6f112a-49fb-4edf-af39-8603398fab38-tmp\") pod \"kube-auth-proxy-799dfc8bc6-flvvx\" (UID: \"3f6f112a-49fb-4edf-af39-8603398fab38\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx"
Sep 09 18:41:32.597529 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.597464    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6f112a-49fb-4edf-af39-8603398fab38-tls-certs\") pod \"kube-auth-proxy-799dfc8bc6-flvvx\" (UID: \"3f6f112a-49fb-4edf-af39-8603398fab38\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx"
Sep 09 18:41:32.597742 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.597562    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xjsf\" (UniqueName: \"kubernetes.io/projected/3f6f112a-49fb-4edf-af39-8603398fab38-kube-api-access-9xjsf\") pod \"kube-auth-proxy-799dfc8bc6-flvvx\" (UID: \"3f6f112a-49fb-4edf-af39-8603398fab38\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx"
Sep 09 18:41:32.597742 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.597584    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f6f112a-49fb-4edf-af39-8603398fab38-tmp\") pod \"kube-auth-proxy-799dfc8bc6-flvvx\" (UID: \"3f6f112a-49fb-4edf-af39-8603398fab38\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx"
Sep 09 18:41:32.599994 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.599961    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f6f112a-49fb-4edf-af39-8603398fab38-tmp\") pod \"kube-auth-proxy-799dfc8bc6-flvvx\" (UID: \"3f6f112a-49fb-4edf-af39-8603398fab38\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx"
Sep 09 18:41:32.600149 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.600136    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6f112a-49fb-4edf-af39-8603398fab38-tls-certs\") pod \"kube-auth-proxy-799dfc8bc6-flvvx\" (UID: \"3f6f112a-49fb-4edf-af39-8603398fab38\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx"
Sep 09 18:41:32.608635 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.608605    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xjsf\" (UniqueName: \"kubernetes.io/projected/3f6f112a-49fb-4edf-af39-8603398fab38-kube-api-access-9xjsf\") pod \"kube-auth-proxy-799dfc8bc6-flvvx\" (UID: \"3f6f112a-49fb-4edf-af39-8603398fab38\") " pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx"
Sep 09 18:41:32.741804 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:32.741718    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx"
Sep 09 18:41:33.074035 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:41:33.074001    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6f112a_49fb_4edf_af39_8603398fab38.slice/crio-712c2957e9833e61056be761ca389e2cd83b7de270af78b42d03db419e65095a WatchSource:0}: Error finding container 712c2957e9833e61056be761ca389e2cd83b7de270af78b42d03db419e65095a: Status 404 returned error can't find the container with id 712c2957e9833e61056be761ca389e2cd83b7de270af78b42d03db419e65095a
Sep 09 18:41:33.074536 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:33.074467    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx"]
Sep 09 18:41:33.623204 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:33.623162    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx" event={"ID":"3f6f112a-49fb-4edf-af39-8603398fab38","Type":"ContainerStarted","Data":"712c2957e9833e61056be761ca389e2cd83b7de270af78b42d03db419e65095a"}
Sep 09 18:41:36.641137 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:36.641098    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx" event={"ID":"3f6f112a-49fb-4edf-af39-8603398fab38","Type":"ContainerStarted","Data":"8c7d9dbab9ae0d9215a78e9e0518008e9aa435885b675ce93df4f5cb6c77f1c0"}
Sep 09 18:41:36.692742 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:36.692683    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-799dfc8bc6-flvvx" podStartSLOduration=1.232403489 podStartE2EDuration="4.692665626s" totalImagesPullingTime="3.460262137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:41:36.691435862 +0000 UTC m=+584.928805679" watchObservedRunningTime="2026-09-09 18:41:36.692665626 +0000 UTC m=+584.930035444"
Sep 09 18:41:38.058926 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.058889    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"]
Sep 09 18:41:38.062373 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.062354    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:38.069184 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.068935    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 18:41:38.069184 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.069063    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 18:41:38.069335 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.069189    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 18:41:38.069335 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.069189    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 18:41:38.069335 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.069226    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-r8d4c\""
Sep 09 18:41:38.069335 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.069330    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 18:41:38.078819 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.078783    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"]
Sep 09 18:41:38.148920 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.148882    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1732ed2-a6c1-443b-8fae-92cc9b4542ea-cert\") pod \"lws-controller-manager-96799d79b-sfldg\" (UID: \"a1732ed2-a6c1-443b-8fae-92cc9b4542ea\") " pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:38.149107 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.148932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn464\" (UniqueName: \"kubernetes.io/projected/a1732ed2-a6c1-443b-8fae-92cc9b4542ea-kube-api-access-kn464\") pod \"lws-controller-manager-96799d79b-sfldg\" (UID: \"a1732ed2-a6c1-443b-8fae-92cc9b4542ea\") " pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:38.149107 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.149027    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1732ed2-a6c1-443b-8fae-92cc9b4542ea-metrics-cert\") pod \"lws-controller-manager-96799d79b-sfldg\" (UID: \"a1732ed2-a6c1-443b-8fae-92cc9b4542ea\") " pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:38.149107 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.149068    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a1732ed2-a6c1-443b-8fae-92cc9b4542ea-manager-config\") pod \"lws-controller-manager-96799d79b-sfldg\" (UID: \"a1732ed2-a6c1-443b-8fae-92cc9b4542ea\") " pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:38.250320 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.250282    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1732ed2-a6c1-443b-8fae-92cc9b4542ea-cert\") pod \"lws-controller-manager-96799d79b-sfldg\" (UID: \"a1732ed2-a6c1-443b-8fae-92cc9b4542ea\") " pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:38.250320 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.250326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn464\" (UniqueName: \"kubernetes.io/projected/a1732ed2-a6c1-443b-8fae-92cc9b4542ea-kube-api-access-kn464\") pod \"lws-controller-manager-96799d79b-sfldg\" (UID: \"a1732ed2-a6c1-443b-8fae-92cc9b4542ea\") " pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:38.250579 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.250366    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1732ed2-a6c1-443b-8fae-92cc9b4542ea-metrics-cert\") pod \"lws-controller-manager-96799d79b-sfldg\" (UID: \"a1732ed2-a6c1-443b-8fae-92cc9b4542ea\") " pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:38.250579 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.250401    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a1732ed2-a6c1-443b-8fae-92cc9b4542ea-manager-config\") pod \"lws-controller-manager-96799d79b-sfldg\" (UID: \"a1732ed2-a6c1-443b-8fae-92cc9b4542ea\") " pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:38.251122 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.251100    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a1732ed2-a6c1-443b-8fae-92cc9b4542ea-manager-config\") pod \"lws-controller-manager-96799d79b-sfldg\" (UID: \"a1732ed2-a6c1-443b-8fae-92cc9b4542ea\") " pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:38.253036 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.253015    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1732ed2-a6c1-443b-8fae-92cc9b4542ea-cert\") pod \"lws-controller-manager-96799d79b-sfldg\" (UID: \"a1732ed2-a6c1-443b-8fae-92cc9b4542ea\") " pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:38.253133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.253064    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1732ed2-a6c1-443b-8fae-92cc9b4542ea-metrics-cert\") pod \"lws-controller-manager-96799d79b-sfldg\" (UID: \"a1732ed2-a6c1-443b-8fae-92cc9b4542ea\") " pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:38.263254 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.263225    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn464\" (UniqueName: \"kubernetes.io/projected/a1732ed2-a6c1-443b-8fae-92cc9b4542ea-kube-api-access-kn464\") pod \"lws-controller-manager-96799d79b-sfldg\" (UID: \"a1732ed2-a6c1-443b-8fae-92cc9b4542ea\") " pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:38.372203 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.372167    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:38.654689 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:38.654659    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"]
Sep 09 18:41:38.656551 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:41:38.656521    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1732ed2_a6c1_443b_8fae_92cc9b4542ea.slice/crio-e33be41d50507c881800de1a097b1a9255c15b39a1eb828f9f2113b1cc1af02e WatchSource:0}: Error finding container e33be41d50507c881800de1a097b1a9255c15b39a1eb828f9f2113b1cc1af02e: Status 404 returned error can't find the container with id e33be41d50507c881800de1a097b1a9255c15b39a1eb828f9f2113b1cc1af02e
Sep 09 18:41:39.657379 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:39.657333    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg" event={"ID":"a1732ed2-a6c1-443b-8fae-92cc9b4542ea","Type":"ContainerStarted","Data":"e33be41d50507c881800de1a097b1a9255c15b39a1eb828f9f2113b1cc1af02e"}
Sep 09 18:41:42.675392 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:42.675348    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg" event={"ID":"a1732ed2-a6c1-443b-8fae-92cc9b4542ea","Type":"ContainerStarted","Data":"efccfbdec9fc8c670239671738309c57256fdddf24987a55cec24b68266a26de"}
Sep 09 18:41:42.675849 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:42.675513    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:41:42.802658 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:42.802591    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg" podStartSLOduration=1.8147672240000001 podStartE2EDuration="4.802568447s" totalImagesPullingTime="2.987801223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:41:42.800782168 +0000 UTC m=+591.038151984" watchObservedRunningTime="2026-09-09 18:41:42.802568447 +0000 UTC m=+591.039938263"
Sep 09 18:41:52.228689 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:52.228659    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 18:41:52.229162 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:52.228990    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 18:41:53.683179 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:41:53.683140    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-96799d79b-sfldg"
Sep 09 18:43:15.465298 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:15.465261    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"]
Sep 09 18:43:15.467648 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:15.467631    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:15.472150 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:15.472124    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5jz4q\""
Sep 09 18:43:15.473235 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:15.473212    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 18:43:15.473319 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:15.473237    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 18:43:15.496038 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:15.496006    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"]
Sep 09 18:43:15.566054 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:15.566006    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lhlq\" (UniqueName: \"kubernetes.io/projected/df317863-9896-4f8f-8f6f-c35854c4f78d-kube-api-access-5lhlq\") pod \"limitador-operator-controller-manager-74495c69f8-fv2mq\" (UID: \"df317863-9896-4f8f-8f6f-c35854c4f78d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:15.667255 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:15.667212    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lhlq\" (UniqueName: \"kubernetes.io/projected/df317863-9896-4f8f-8f6f-c35854c4f78d-kube-api-access-5lhlq\") pod \"limitador-operator-controller-manager-74495c69f8-fv2mq\" (UID: \"df317863-9896-4f8f-8f6f-c35854c4f78d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:15.693684 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:15.693646    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lhlq\" (UniqueName: \"kubernetes.io/projected/df317863-9896-4f8f-8f6f-c35854c4f78d-kube-api-access-5lhlq\") pod \"limitador-operator-controller-manager-74495c69f8-fv2mq\" (UID: \"df317863-9896-4f8f-8f6f-c35854c4f78d\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:15.778130 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:15.777961    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:16.078147 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:43:16.078113    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf317863_9896_4f8f_8f6f_c35854c4f78d.slice/crio-93de330893bdccf106789b8345782a4382f15c454a147598b5c6c7a1d9c7f7a2 WatchSource:0}: Error finding container 93de330893bdccf106789b8345782a4382f15c454a147598b5c6c7a1d9c7f7a2: Status 404 returned error can't find the container with id 93de330893bdccf106789b8345782a4382f15c454a147598b5c6c7a1d9c7f7a2
Sep 09 18:43:16.080954 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:16.080933    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 18:43:16.086077 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:16.086044    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"]
Sep 09 18:43:16.121108 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:16.121076    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq" event={"ID":"df317863-9896-4f8f-8f6f-c35854c4f78d","Type":"ContainerStarted","Data":"93de330893bdccf106789b8345782a4382f15c454a147598b5c6c7a1d9c7f7a2"}
Sep 09 18:43:19.138513 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:19.138474    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq" event={"ID":"df317863-9896-4f8f-8f6f-c35854c4f78d","Type":"ContainerStarted","Data":"36fa48aeb8b6a081d9a1c963f8c5b689fc473d83ab51bc9520a5f840c1b4b0ed"}
Sep 09 18:43:19.138924 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:19.138680    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:19.171291 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:19.171187    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq" podStartSLOduration=1.343869066 podStartE2EDuration="4.171170333s" totalImagesPullingTime="2.827301267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:43:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:43:19.170941263 +0000 UTC m=+687.408311079" watchObservedRunningTime="2026-09-09 18:43:19.171170333 +0000 UTC m=+687.408540148"
Sep 09 18:43:27.448268 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.448226    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"]
Sep 09 18:43:27.448868 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.448436    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq" podUID="df317863-9896-4f8f-8f6f-c35854c4f78d" containerName="manager" containerID="cri-o://36fa48aeb8b6a081d9a1c963f8c5b689fc473d83ab51bc9520a5f840c1b4b0ed" gracePeriod=2
Sep 09 18:43:27.450878 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.450843    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:27.502301 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.502268    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"]
Sep 09 18:43:27.517889 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:43:27.517846    2575 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"df317863-9896-4f8f-8f6f-c35854c4f78d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-fv2mq\": pods \"limitador-operator-controller-manager-74495c69f8-fv2mq\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:27.520264 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:43:27.520232    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-fv2mq\" is forbidden: User \"system:node:ip-10-0-134-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-63.ec2.internal' and this object" podUID="df317863-9896-4f8f-8f6f-c35854c4f78d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:27.563138 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.563105    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-djd9z"]
Sep 09 18:43:27.563577 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.563541    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df317863-9896-4f8f-8f6f-c35854c4f78d" containerName="manager"
Sep 09 18:43:27.563577 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.563558    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="df317863-9896-4f8f-8f6f-c35854c4f78d" containerName="manager"
Sep 09 18:43:27.563704 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.563666    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="df317863-9896-4f8f-8f6f-c35854c4f78d" containerName="manager"
Sep 09 18:43:27.565818 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.565795    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-djd9z"
Sep 09 18:43:27.573671 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:43:27.573617    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-fv2mq\" is forbidden: User \"system:node:ip-10-0-134-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-63.ec2.internal' and this object" podUID="df317863-9896-4f8f-8f6f-c35854c4f78d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:27.576000 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.575968    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmvsr\" (UniqueName: \"kubernetes.io/projected/d5d8952a-3efc-444c-b2fb-b41ee743bc67-kube-api-access-bmvsr\") pod \"limitador-operator-controller-manager-74495c69f8-djd9z\" (UID: \"d5d8952a-3efc-444c-b2fb-b41ee743bc67\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-djd9z"
Sep 09 18:43:27.590024 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.589989    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-djd9z"]
Sep 09 18:43:27.677322 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.677286    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmvsr\" (UniqueName: \"kubernetes.io/projected/d5d8952a-3efc-444c-b2fb-b41ee743bc67-kube-api-access-bmvsr\") pod \"limitador-operator-controller-manager-74495c69f8-djd9z\" (UID: \"d5d8952a-3efc-444c-b2fb-b41ee743bc67\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-djd9z"
Sep 09 18:43:27.717759 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.717718    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmvsr\" (UniqueName: \"kubernetes.io/projected/d5d8952a-3efc-444c-b2fb-b41ee743bc67-kube-api-access-bmvsr\") pod \"limitador-operator-controller-manager-74495c69f8-djd9z\" (UID: \"d5d8952a-3efc-444c-b2fb-b41ee743bc67\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-djd9z"
Sep 09 18:43:27.733550 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.733522    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:27.736373 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:43:27.736343    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-fv2mq\" is forbidden: User \"system:node:ip-10-0-134-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-63.ec2.internal' and this object" podUID="df317863-9896-4f8f-8f6f-c35854c4f78d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:27.777995 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.777956    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lhlq\" (UniqueName: \"kubernetes.io/projected/df317863-9896-4f8f-8f6f-c35854c4f78d-kube-api-access-5lhlq\") pod \"df317863-9896-4f8f-8f6f-c35854c4f78d\" (UID: \"df317863-9896-4f8f-8f6f-c35854c4f78d\") "
Sep 09 18:43:27.780311 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.780282    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df317863-9896-4f8f-8f6f-c35854c4f78d-kube-api-access-5lhlq" (OuterVolumeSpecName: "kube-api-access-5lhlq") pod "df317863-9896-4f8f-8f6f-c35854c4f78d" (UID: "df317863-9896-4f8f-8f6f-c35854c4f78d"). InnerVolumeSpecName "kube-api-access-5lhlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:43:27.878046 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.878006    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-djd9z"
Sep 09 18:43:27.879326 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:27.879300    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lhlq\" (UniqueName: \"kubernetes.io/projected/df317863-9896-4f8f-8f6f-c35854c4f78d-kube-api-access-5lhlq\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:43:28.182706 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:28.182677    2575 generic.go:356] "Generic (PLEG): container finished" podID="df317863-9896-4f8f-8f6f-c35854c4f78d" containerID="36fa48aeb8b6a081d9a1c963f8c5b689fc473d83ab51bc9520a5f840c1b4b0ed" exitCode=0
Sep 09 18:43:28.182801 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:28.182727    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:28.182801 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:28.182773    2575 scope.go:117] "RemoveContainer" containerID="36fa48aeb8b6a081d9a1c963f8c5b689fc473d83ab51bc9520a5f840c1b4b0ed"
Sep 09 18:43:28.185019 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:43:28.184989    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d8952a_3efc_444c_b2fb_b41ee743bc67.slice/crio-82777e81d6594eb8711c6b2795193ec0e01c7cec6f730b7adcfd9abf894517a5 WatchSource:0}: Error finding container 82777e81d6594eb8711c6b2795193ec0e01c7cec6f730b7adcfd9abf894517a5: Status 404 returned error can't find the container with id 82777e81d6594eb8711c6b2795193ec0e01c7cec6f730b7adcfd9abf894517a5
Sep 09 18:43:28.185868 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:43:28.185835    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-fv2mq\" is forbidden: User \"system:node:ip-10-0-134-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-63.ec2.internal' and this object" podUID="df317863-9896-4f8f-8f6f-c35854c4f78d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:28.191226 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:28.191195    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-djd9z"]
Sep 09 18:43:28.192922 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:28.192885    2575 scope.go:117] "RemoveContainer" containerID="36fa48aeb8b6a081d9a1c963f8c5b689fc473d83ab51bc9520a5f840c1b4b0ed"
Sep 09 18:43:28.193336 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:43:28.193291    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36fa48aeb8b6a081d9a1c963f8c5b689fc473d83ab51bc9520a5f840c1b4b0ed\": container with ID starting with 36fa48aeb8b6a081d9a1c963f8c5b689fc473d83ab51bc9520a5f840c1b4b0ed not found: ID does not exist" containerID="36fa48aeb8b6a081d9a1c963f8c5b689fc473d83ab51bc9520a5f840c1b4b0ed"
Sep 09 18:43:28.193431 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:28.193342    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36fa48aeb8b6a081d9a1c963f8c5b689fc473d83ab51bc9520a5f840c1b4b0ed"} err="failed to get container status \"36fa48aeb8b6a081d9a1c963f8c5b689fc473d83ab51bc9520a5f840c1b4b0ed\": rpc error: code = NotFound desc = could not find container \"36fa48aeb8b6a081d9a1c963f8c5b689fc473d83ab51bc9520a5f840c1b4b0ed\": container with ID starting with 36fa48aeb8b6a081d9a1c963f8c5b689fc473d83ab51bc9520a5f840c1b4b0ed not found: ID does not exist"
Sep 09 18:43:28.200034 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:43:28.199997    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-fv2mq\" is forbidden: User \"system:node:ip-10-0-134-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-63.ec2.internal' and this object" podUID="df317863-9896-4f8f-8f6f-c35854c4f78d" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-fv2mq"
Sep 09 18:43:28.321262 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:28.321222    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df317863-9896-4f8f-8f6f-c35854c4f78d" path="/var/lib/kubelet/pods/df317863-9896-4f8f-8f6f-c35854c4f78d/volumes"
Sep 09 18:43:29.189340 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:29.189300    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-djd9z" event={"ID":"d5d8952a-3efc-444c-b2fb-b41ee743bc67","Type":"ContainerStarted","Data":"4013fa2a0ce0dccae9af06b6bb2797a7c9376e9082b295c8f389fda4bd966226"}
Sep 09 18:43:29.189818 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:29.189360    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-djd9z" event={"ID":"d5d8952a-3efc-444c-b2fb-b41ee743bc67","Type":"ContainerStarted","Data":"82777e81d6594eb8711c6b2795193ec0e01c7cec6f730b7adcfd9abf894517a5"}
Sep 09 18:43:29.189818 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:29.189386    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-djd9z"
Sep 09 18:43:29.249590 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:29.249533    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-djd9z" podStartSLOduration=2.24951215 podStartE2EDuration="2.24951215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:43:29.235832148 +0000 UTC m=+697.473201956" watchObservedRunningTime="2026-09-09 18:43:29.24951215 +0000 UTC m=+697.486881956"
Sep 09 18:43:40.197431 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:43:40.197394    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-djd9z"
Sep 09 18:44:04.017600 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.017562    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qxlhb"]
Sep 09 18:44:04.024281 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.024257    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qxlhb"
Sep 09 18:44:04.027162 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.027133    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9bfhq\""
Sep 09 18:44:04.048315 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.048282    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qxlhb"]
Sep 09 18:44:04.117029 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.116988    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqjzt\" (UniqueName: \"kubernetes.io/projected/30a1dcec-eba0-4824-8ccc-f04856bfecaa-kube-api-access-tqjzt\") pod \"authorino-7bcfd8f9cd-qxlhb\" (UID: \"30a1dcec-eba0-4824-8ccc-f04856bfecaa\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qxlhb"
Sep 09 18:44:04.218315 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.218278    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqjzt\" (UniqueName: \"kubernetes.io/projected/30a1dcec-eba0-4824-8ccc-f04856bfecaa-kube-api-access-tqjzt\") pod \"authorino-7bcfd8f9cd-qxlhb\" (UID: \"30a1dcec-eba0-4824-8ccc-f04856bfecaa\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qxlhb"
Sep 09 18:44:04.231516 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.231465    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqjzt\" (UniqueName: \"kubernetes.io/projected/30a1dcec-eba0-4824-8ccc-f04856bfecaa-kube-api-access-tqjzt\") pod \"authorino-7bcfd8f9cd-qxlhb\" (UID: \"30a1dcec-eba0-4824-8ccc-f04856bfecaa\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qxlhb"
Sep 09 18:44:04.313482 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.313397    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-2c72t"]
Sep 09 18:44:04.316862 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.316837    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2c72t"
Sep 09 18:44:04.319524 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.319488    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vq9k\" (UniqueName: \"kubernetes.io/projected/71c4473e-a8fc-4d37-9b3e-02c27738d636-kube-api-access-2vq9k\") pod \"authorino-59c974bd48-2c72t\" (UID: \"71c4473e-a8fc-4d37-9b3e-02c27738d636\") " pod="kuadrant-system/authorino-59c974bd48-2c72t"
Sep 09 18:44:04.340321 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.340285    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2c72t"]
Sep 09 18:44:04.350220 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.350188    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qxlhb"
Sep 09 18:44:04.420996 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.420336    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vq9k\" (UniqueName: \"kubernetes.io/projected/71c4473e-a8fc-4d37-9b3e-02c27738d636-kube-api-access-2vq9k\") pod \"authorino-59c974bd48-2c72t\" (UID: \"71c4473e-a8fc-4d37-9b3e-02c27738d636\") " pod="kuadrant-system/authorino-59c974bd48-2c72t"
Sep 09 18:44:04.432613 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.432578    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vq9k\" (UniqueName: \"kubernetes.io/projected/71c4473e-a8fc-4d37-9b3e-02c27738d636-kube-api-access-2vq9k\") pod \"authorino-59c974bd48-2c72t\" (UID: \"71c4473e-a8fc-4d37-9b3e-02c27738d636\") " pod="kuadrant-system/authorino-59c974bd48-2c72t"
Sep 09 18:44:04.627818 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.627772    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2c72t"
Sep 09 18:44:04.631258 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.631074    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qxlhb"]
Sep 09 18:44:04.634217 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:44:04.634187    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30a1dcec_eba0_4824_8ccc_f04856bfecaa.slice/crio-80904afd6fb392dc37e7b500cd1eff9452cf1f99f7d978e2352150ccb464e6dd WatchSource:0}: Error finding container 80904afd6fb392dc37e7b500cd1eff9452cf1f99f7d978e2352150ccb464e6dd: Status 404 returned error can't find the container with id 80904afd6fb392dc37e7b500cd1eff9452cf1f99f7d978e2352150ccb464e6dd
Sep 09 18:44:04.912389 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:04.912306    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2c72t"]
Sep 09 18:44:04.918166 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:44:04.918129    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71c4473e_a8fc_4d37_9b3e_02c27738d636.slice/crio-b08256db4629fc7cd83997d5e3210419ef8c496420977d0c59480f86b3af82d6 WatchSource:0}: Error finding container b08256db4629fc7cd83997d5e3210419ef8c496420977d0c59480f86b3af82d6: Status 404 returned error can't find the container with id b08256db4629fc7cd83997d5e3210419ef8c496420977d0c59480f86b3af82d6
Sep 09 18:44:05.366570 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:05.366528    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2c72t" event={"ID":"71c4473e-a8fc-4d37-9b3e-02c27738d636","Type":"ContainerStarted","Data":"b08256db4629fc7cd83997d5e3210419ef8c496420977d0c59480f86b3af82d6"}
Sep 09 18:44:05.367764 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:05.367735    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qxlhb" event={"ID":"30a1dcec-eba0-4824-8ccc-f04856bfecaa","Type":"ContainerStarted","Data":"80904afd6fb392dc37e7b500cd1eff9452cf1f99f7d978e2352150ccb464e6dd"}
Sep 09 18:44:07.381008 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:07.380963    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2c72t" event={"ID":"71c4473e-a8fc-4d37-9b3e-02c27738d636","Type":"ContainerStarted","Data":"686716d8001f5fdb1114304239674d5e585a880ff10ab056867957bec805e89a"}
Sep 09 18:44:07.382850 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:07.382819    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qxlhb" event={"ID":"30a1dcec-eba0-4824-8ccc-f04856bfecaa","Type":"ContainerStarted","Data":"15d9c42a80b71546a26bfdaa04de8580f59af63352dce3fd46a8f44c9eb88d46"}
Sep 09 18:44:07.399313 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:07.399243    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-2c72t" podStartSLOduration=1.059204328 podStartE2EDuration="3.399217055s" totalImagesPullingTime="2.340012727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:44:07.397258987 +0000 UTC m=+735.634628802" watchObservedRunningTime="2026-09-09 18:44:07.399217055 +0000 UTC m=+735.636586872"
Sep 09 18:44:07.453187 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:07.453117    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-qxlhb" podStartSLOduration=1.824377246 podStartE2EDuration="4.453089458s" totalImagesPullingTime="2.628712212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:44:07.419942531 +0000 UTC m=+735.657312345" watchObservedRunningTime="2026-09-09 18:44:07.453089458 +0000 UTC m=+735.690459275"
Sep 09 18:44:07.453401 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:07.453272    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qxlhb"]
Sep 09 18:44:09.392580 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:09.392510    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-qxlhb" podUID="30a1dcec-eba0-4824-8ccc-f04856bfecaa" containerName="authorino" containerID="cri-o://15d9c42a80b71546a26bfdaa04de8580f59af63352dce3fd46a8f44c9eb88d46" gracePeriod=30
Sep 09 18:44:09.647426 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:09.647345    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qxlhb"
Sep 09 18:44:09.664387 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:09.664347    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqjzt\" (UniqueName: \"kubernetes.io/projected/30a1dcec-eba0-4824-8ccc-f04856bfecaa-kube-api-access-tqjzt\") pod \"30a1dcec-eba0-4824-8ccc-f04856bfecaa\" (UID: \"30a1dcec-eba0-4824-8ccc-f04856bfecaa\") "
Sep 09 18:44:09.666802 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:09.666756    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a1dcec-eba0-4824-8ccc-f04856bfecaa-kube-api-access-tqjzt" (OuterVolumeSpecName: "kube-api-access-tqjzt") pod "30a1dcec-eba0-4824-8ccc-f04856bfecaa" (UID: "30a1dcec-eba0-4824-8ccc-f04856bfecaa"). InnerVolumeSpecName "kube-api-access-tqjzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:44:09.765457 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:09.765408    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqjzt\" (UniqueName: \"kubernetes.io/projected/30a1dcec-eba0-4824-8ccc-f04856bfecaa-kube-api-access-tqjzt\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:44:10.399784 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:10.399750    2575 generic.go:356] "Generic (PLEG): container finished" podID="30a1dcec-eba0-4824-8ccc-f04856bfecaa" containerID="15d9c42a80b71546a26bfdaa04de8580f59af63352dce3fd46a8f44c9eb88d46" exitCode=0
Sep 09 18:44:10.399784 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:10.399792    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qxlhb"
Sep 09 18:44:10.400307 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:10.399821    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qxlhb" event={"ID":"30a1dcec-eba0-4824-8ccc-f04856bfecaa","Type":"ContainerDied","Data":"15d9c42a80b71546a26bfdaa04de8580f59af63352dce3fd46a8f44c9eb88d46"}
Sep 09 18:44:10.400307 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:10.399854    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qxlhb" event={"ID":"30a1dcec-eba0-4824-8ccc-f04856bfecaa","Type":"ContainerDied","Data":"80904afd6fb392dc37e7b500cd1eff9452cf1f99f7d978e2352150ccb464e6dd"}
Sep 09 18:44:10.400307 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:10.399872    2575 scope.go:117] "RemoveContainer" containerID="15d9c42a80b71546a26bfdaa04de8580f59af63352dce3fd46a8f44c9eb88d46"
Sep 09 18:44:10.409017 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:10.408995    2575 scope.go:117] "RemoveContainer" containerID="15d9c42a80b71546a26bfdaa04de8580f59af63352dce3fd46a8f44c9eb88d46"
Sep 09 18:44:10.409378 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:44:10.409357    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d9c42a80b71546a26bfdaa04de8580f59af63352dce3fd46a8f44c9eb88d46\": container with ID starting with 15d9c42a80b71546a26bfdaa04de8580f59af63352dce3fd46a8f44c9eb88d46 not found: ID does not exist" containerID="15d9c42a80b71546a26bfdaa04de8580f59af63352dce3fd46a8f44c9eb88d46"
Sep 09 18:44:10.409429 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:10.409387    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d9c42a80b71546a26bfdaa04de8580f59af63352dce3fd46a8f44c9eb88d46"} err="failed to get container status \"15d9c42a80b71546a26bfdaa04de8580f59af63352dce3fd46a8f44c9eb88d46\": rpc error: code = NotFound desc = could not find container \"15d9c42a80b71546a26bfdaa04de8580f59af63352dce3fd46a8f44c9eb88d46\": container with ID starting with 15d9c42a80b71546a26bfdaa04de8580f59af63352dce3fd46a8f44c9eb88d46 not found: ID does not exist"
Sep 09 18:44:10.423721 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:10.423685    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qxlhb"]
Sep 09 18:44:10.428989 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:10.428956    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qxlhb"]
Sep 09 18:44:12.320812 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:12.320779    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30a1dcec-eba0-4824-8ccc-f04856bfecaa" path="/var/lib/kubelet/pods/30a1dcec-eba0-4824-8ccc-f04856bfecaa/volumes"
Sep 09 18:44:32.620062 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.620019    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-czqnm"]
Sep 09 18:44:32.620452 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.620379    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a1dcec-eba0-4824-8ccc-f04856bfecaa" containerName="authorino"
Sep 09 18:44:32.620452 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.620391    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a1dcec-eba0-4824-8ccc-f04856bfecaa" containerName="authorino"
Sep 09 18:44:32.620623 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.620463    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a1dcec-eba0-4824-8ccc-f04856bfecaa" containerName="authorino"
Sep 09 18:44:32.623382 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.623362    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-czqnm"
Sep 09 18:44:32.632287 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.632254    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-czqnm"]
Sep 09 18:44:32.705392 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.705357    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-czqnm"]
Sep 09 18:44:32.705704 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:44:32.705683    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4kt27], unattached volumes=[], failed to process volumes=[kube-api-access-4kt27]: context canceled" pod="kuadrant-system/authorino-557f876f69-czqnm" podUID="aaf50c8b-365a-4129-898c-92c031497145"
Sep 09 18:44:32.745196 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.745155    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-lqknn"]
Sep 09 18:44:32.748591 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.748569    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-lqknn"
Sep 09 18:44:32.752204 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.752181    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 18:44:32.759237 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.759206    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-lqknn"]
Sep 09 18:44:32.785579 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.782026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kt27\" (UniqueName: \"kubernetes.io/projected/aaf50c8b-365a-4129-898c-92c031497145-kube-api-access-4kt27\") pod \"authorino-557f876f69-czqnm\" (UID: \"aaf50c8b-365a-4129-898c-92c031497145\") " pod="kuadrant-system/authorino-557f876f69-czqnm"
Sep 09 18:44:32.878328 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.878227    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-lqknn"]
Sep 09 18:44:32.878582 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:44:32.878557    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jrsvt tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-lqknn" podUID="4ad5a6cd-d3e7-4a0d-a741-adabfa864edd"
Sep 09 18:44:32.883418 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.883383    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrsvt\" (UniqueName: \"kubernetes.io/projected/4ad5a6cd-d3e7-4a0d-a741-adabfa864edd-kube-api-access-jrsvt\") pod \"authorino-6d4dfc649f-lqknn\" (UID: \"4ad5a6cd-d3e7-4a0d-a741-adabfa864edd\") " pod="kuadrant-system/authorino-6d4dfc649f-lqknn"
Sep 09 18:44:32.883563 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.883436    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4ad5a6cd-d3e7-4a0d-a741-adabfa864edd-tls-cert\") pod \"authorino-6d4dfc649f-lqknn\" (UID: \"4ad5a6cd-d3e7-4a0d-a741-adabfa864edd\") " pod="kuadrant-system/authorino-6d4dfc649f-lqknn"
Sep 09 18:44:32.883563 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.883464    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kt27\" (UniqueName: \"kubernetes.io/projected/aaf50c8b-365a-4129-898c-92c031497145-kube-api-access-4kt27\") pod \"authorino-557f876f69-czqnm\" (UID: \"aaf50c8b-365a-4129-898c-92c031497145\") " pod="kuadrant-system/authorino-557f876f69-czqnm"
Sep 09 18:44:32.902902 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.902863    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kt27\" (UniqueName: \"kubernetes.io/projected/aaf50c8b-365a-4129-898c-92c031497145-kube-api-access-4kt27\") pod \"authorino-557f876f69-czqnm\" (UID: \"aaf50c8b-365a-4129-898c-92c031497145\") " pod="kuadrant-system/authorino-557f876f69-czqnm"
Sep 09 18:44:32.912204 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.912167    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c8df6648-twlfz"]
Sep 09 18:44:32.915676 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.915649    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c8df6648-twlfz"
Sep 09 18:44:32.930484 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.930456    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c8df6648-twlfz"]
Sep 09 18:44:32.984159 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.984111    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsvt\" (UniqueName: \"kubernetes.io/projected/4ad5a6cd-d3e7-4a0d-a741-adabfa864edd-kube-api-access-jrsvt\") pod \"authorino-6d4dfc649f-lqknn\" (UID: \"4ad5a6cd-d3e7-4a0d-a741-adabfa864edd\") " pod="kuadrant-system/authorino-6d4dfc649f-lqknn"
Sep 09 18:44:32.984340 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.984173    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4ad5a6cd-d3e7-4a0d-a741-adabfa864edd-tls-cert\") pod \"authorino-6d4dfc649f-lqknn\" (UID: \"4ad5a6cd-d3e7-4a0d-a741-adabfa864edd\") " pod="kuadrant-system/authorino-6d4dfc649f-lqknn"
Sep 09 18:44:32.986803 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.986775    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4ad5a6cd-d3e7-4a0d-a741-adabfa864edd-tls-cert\") pod \"authorino-6d4dfc649f-lqknn\" (UID: \"4ad5a6cd-d3e7-4a0d-a741-adabfa864edd\") " pod="kuadrant-system/authorino-6d4dfc649f-lqknn"
Sep 09 18:44:32.992994 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:32.992962    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrsvt\" (UniqueName: \"kubernetes.io/projected/4ad5a6cd-d3e7-4a0d-a741-adabfa864edd-kube-api-access-jrsvt\") pod \"authorino-6d4dfc649f-lqknn\" (UID: \"4ad5a6cd-d3e7-4a0d-a741-adabfa864edd\") " pod="kuadrant-system/authorino-6d4dfc649f-lqknn"
Sep 09 18:44:33.084760 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.084707    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpm5j\" (UniqueName: \"kubernetes.io/projected/0553582f-36c0-4143-baf9-579ca80ab2b1-kube-api-access-hpm5j\") pod \"authorino-6c8df6648-twlfz\" (UID: \"0553582f-36c0-4143-baf9-579ca80ab2b1\") " pod="kuadrant-system/authorino-6c8df6648-twlfz"
Sep 09 18:44:33.084927 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.084837    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0553582f-36c0-4143-baf9-579ca80ab2b1-tls-cert\") pod \"authorino-6c8df6648-twlfz\" (UID: \"0553582f-36c0-4143-baf9-579ca80ab2b1\") " pod="kuadrant-system/authorino-6c8df6648-twlfz"
Sep 09 18:44:33.185451 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.185348    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpm5j\" (UniqueName: \"kubernetes.io/projected/0553582f-36c0-4143-baf9-579ca80ab2b1-kube-api-access-hpm5j\") pod \"authorino-6c8df6648-twlfz\" (UID: \"0553582f-36c0-4143-baf9-579ca80ab2b1\") " pod="kuadrant-system/authorino-6c8df6648-twlfz"
Sep 09 18:44:33.185451 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.185441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0553582f-36c0-4143-baf9-579ca80ab2b1-tls-cert\") pod \"authorino-6c8df6648-twlfz\" (UID: \"0553582f-36c0-4143-baf9-579ca80ab2b1\") " pod="kuadrant-system/authorino-6c8df6648-twlfz"
Sep 09 18:44:33.188010 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.187985    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0553582f-36c0-4143-baf9-579ca80ab2b1-tls-cert\") pod \"authorino-6c8df6648-twlfz\" (UID: \"0553582f-36c0-4143-baf9-579ca80ab2b1\") " pod="kuadrant-system/authorino-6c8df6648-twlfz"
Sep 09 18:44:33.196944 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.196914    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpm5j\" (UniqueName: \"kubernetes.io/projected/0553582f-36c0-4143-baf9-579ca80ab2b1-kube-api-access-hpm5j\") pod \"authorino-6c8df6648-twlfz\" (UID: \"0553582f-36c0-4143-baf9-579ca80ab2b1\") " pod="kuadrant-system/authorino-6c8df6648-twlfz"
Sep 09 18:44:33.225915 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.225874    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c8df6648-twlfz"
Sep 09 18:44:33.503170 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.503141    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c8df6648-twlfz"]
Sep 09 18:44:33.505782 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:44:33.505749    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0553582f_36c0_4143_baf9_579ca80ab2b1.slice/crio-eab6a07532f3f216374dee7a55ce43d9b3c2873502bf5cd09db7603193426d4b WatchSource:0}: Error finding container eab6a07532f3f216374dee7a55ce43d9b3c2873502bf5cd09db7603193426d4b: Status 404 returned error can't find the container with id eab6a07532f3f216374dee7a55ce43d9b3c2873502bf5cd09db7603193426d4b
Sep 09 18:44:33.514615 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.514581    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c8df6648-twlfz" event={"ID":"0553582f-36c0-4143-baf9-579ca80ab2b1","Type":"ContainerStarted","Data":"eab6a07532f3f216374dee7a55ce43d9b3c2873502bf5cd09db7603193426d4b"}
Sep 09 18:44:33.514615 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.514608    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-lqknn"
Sep 09 18:44:33.514805 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.514667    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-czqnm"
Sep 09 18:44:33.523848 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.523822    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-lqknn"
Sep 09 18:44:33.526178 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.526155    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-czqnm"
Sep 09 18:44:33.690605 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.690552    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4ad5a6cd-d3e7-4a0d-a741-adabfa864edd-tls-cert\") pod \"4ad5a6cd-d3e7-4a0d-a741-adabfa864edd\" (UID: \"4ad5a6cd-d3e7-4a0d-a741-adabfa864edd\") "
Sep 09 18:44:33.691075 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.690691    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kt27\" (UniqueName: \"kubernetes.io/projected/aaf50c8b-365a-4129-898c-92c031497145-kube-api-access-4kt27\") pod \"aaf50c8b-365a-4129-898c-92c031497145\" (UID: \"aaf50c8b-365a-4129-898c-92c031497145\") "
Sep 09 18:44:33.691075 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.690748    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrsvt\" (UniqueName: \"kubernetes.io/projected/4ad5a6cd-d3e7-4a0d-a741-adabfa864edd-kube-api-access-jrsvt\") pod \"4ad5a6cd-d3e7-4a0d-a741-adabfa864edd\" (UID: \"4ad5a6cd-d3e7-4a0d-a741-adabfa864edd\") "
Sep 09 18:44:33.692789 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.692752    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad5a6cd-d3e7-4a0d-a741-adabfa864edd-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4ad5a6cd-d3e7-4a0d-a741-adabfa864edd" (UID: "4ad5a6cd-d3e7-4a0d-a741-adabfa864edd"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:44:33.692922 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.692860    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf50c8b-365a-4129-898c-92c031497145-kube-api-access-4kt27" (OuterVolumeSpecName: "kube-api-access-4kt27") pod "aaf50c8b-365a-4129-898c-92c031497145" (UID: "aaf50c8b-365a-4129-898c-92c031497145"). InnerVolumeSpecName "kube-api-access-4kt27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:44:33.692922 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.692874    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad5a6cd-d3e7-4a0d-a741-adabfa864edd-kube-api-access-jrsvt" (OuterVolumeSpecName: "kube-api-access-jrsvt") pod "4ad5a6cd-d3e7-4a0d-a741-adabfa864edd" (UID: "4ad5a6cd-d3e7-4a0d-a741-adabfa864edd"). InnerVolumeSpecName "kube-api-access-jrsvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:44:33.791629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.791532    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrsvt\" (UniqueName: \"kubernetes.io/projected/4ad5a6cd-d3e7-4a0d-a741-adabfa864edd-kube-api-access-jrsvt\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:44:33.791629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.791568    2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4ad5a6cd-d3e7-4a0d-a741-adabfa864edd-tls-cert\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:44:33.791629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:33.791578    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kt27\" (UniqueName: \"kubernetes.io/projected/aaf50c8b-365a-4129-898c-92c031497145-kube-api-access-4kt27\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:44:34.522700 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:34.522666    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-lqknn"
Sep 09 18:44:34.522700 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:34.522666    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c8df6648-twlfz" event={"ID":"0553582f-36c0-4143-baf9-579ca80ab2b1","Type":"ContainerStarted","Data":"e760b4d0e5cf6d914a5606e735f0f2e0d96b05ba44691cec132e954f89129813"}
Sep 09 18:44:34.522970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:34.522882    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-czqnm"
Sep 09 18:44:34.543815 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:34.543739    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c8df6648-twlfz" podStartSLOduration=2.006601761 podStartE2EDuration="2.543720693s" totalImagesPullingTime="537.118932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:44:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:44:34.540399256 +0000 UTC m=+762.777769086" watchObservedRunningTime="2026-09-09 18:44:34.543720693 +0000 UTC m=+762.781090508"
Sep 09 18:44:34.574866 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:34.574825    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-lqknn"]
Sep 09 18:44:34.579294 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:34.579193    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-lqknn"]
Sep 09 18:44:34.584208 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:34.584179    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2c72t"]
Sep 09 18:44:34.584421 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:34.584395    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-2c72t" podUID="71c4473e-a8fc-4d37-9b3e-02c27738d636" containerName="authorino" containerID="cri-o://686716d8001f5fdb1114304239674d5e585a880ff10ab056867957bec805e89a" gracePeriod=30
Sep 09 18:44:34.612716 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:34.612682    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-czqnm"]
Sep 09 18:44:34.615572 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:34.615532    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-czqnm"]
Sep 09 18:44:34.822692 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:34.822662    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2c72t"
Sep 09 18:44:34.900999 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:34.900959    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vq9k\" (UniqueName: \"kubernetes.io/projected/71c4473e-a8fc-4d37-9b3e-02c27738d636-kube-api-access-2vq9k\") pod \"71c4473e-a8fc-4d37-9b3e-02c27738d636\" (UID: \"71c4473e-a8fc-4d37-9b3e-02c27738d636\") "
Sep 09 18:44:34.903207 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:34.903167    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c4473e-a8fc-4d37-9b3e-02c27738d636-kube-api-access-2vq9k" (OuterVolumeSpecName: "kube-api-access-2vq9k") pod "71c4473e-a8fc-4d37-9b3e-02c27738d636" (UID: "71c4473e-a8fc-4d37-9b3e-02c27738d636"). InnerVolumeSpecName "kube-api-access-2vq9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:44:35.002251 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:35.002205    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vq9k\" (UniqueName: \"kubernetes.io/projected/71c4473e-a8fc-4d37-9b3e-02c27738d636-kube-api-access-2vq9k\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:44:35.528672 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:35.528637    2575 generic.go:356] "Generic (PLEG): container finished" podID="71c4473e-a8fc-4d37-9b3e-02c27738d636" containerID="686716d8001f5fdb1114304239674d5e585a880ff10ab056867957bec805e89a" exitCode=0
Sep 09 18:44:35.528846 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:35.528676    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2c72t"
Sep 09 18:44:35.528846 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:35.528725    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2c72t" event={"ID":"71c4473e-a8fc-4d37-9b3e-02c27738d636","Type":"ContainerDied","Data":"686716d8001f5fdb1114304239674d5e585a880ff10ab056867957bec805e89a"}
Sep 09 18:44:35.528846 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:35.528767    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2c72t" event={"ID":"71c4473e-a8fc-4d37-9b3e-02c27738d636","Type":"ContainerDied","Data":"b08256db4629fc7cd83997d5e3210419ef8c496420977d0c59480f86b3af82d6"}
Sep 09 18:44:35.528846 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:35.528789    2575 scope.go:117] "RemoveContainer" containerID="686716d8001f5fdb1114304239674d5e585a880ff10ab056867957bec805e89a"
Sep 09 18:44:35.537805 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:35.537785    2575 scope.go:117] "RemoveContainer" containerID="686716d8001f5fdb1114304239674d5e585a880ff10ab056867957bec805e89a"
Sep 09 18:44:35.538125 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:44:35.538106    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686716d8001f5fdb1114304239674d5e585a880ff10ab056867957bec805e89a\": container with ID starting with 686716d8001f5fdb1114304239674d5e585a880ff10ab056867957bec805e89a not found: ID does not exist" containerID="686716d8001f5fdb1114304239674d5e585a880ff10ab056867957bec805e89a"
Sep 09 18:44:35.538191 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:35.538140    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686716d8001f5fdb1114304239674d5e585a880ff10ab056867957bec805e89a"} err="failed to get container status \"686716d8001f5fdb1114304239674d5e585a880ff10ab056867957bec805e89a\": rpc error: code = NotFound desc = could not find container \"686716d8001f5fdb1114304239674d5e585a880ff10ab056867957bec805e89a\": container with ID starting with 686716d8001f5fdb1114304239674d5e585a880ff10ab056867957bec805e89a not found: ID does not exist"
Sep 09 18:44:35.561745 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:35.561711    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2c72t"]
Sep 09 18:44:35.565367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:35.565332    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2c72t"]
Sep 09 18:44:36.320788 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:36.320750    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ad5a6cd-d3e7-4a0d-a741-adabfa864edd" path="/var/lib/kubelet/pods/4ad5a6cd-d3e7-4a0d-a741-adabfa864edd/volumes"
Sep 09 18:44:36.321179 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:36.321020    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71c4473e-a8fc-4d37-9b3e-02c27738d636" path="/var/lib/kubelet/pods/71c4473e-a8fc-4d37-9b3e-02c27738d636/volumes"
Sep 09 18:44:36.321334 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:44:36.321321    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaf50c8b-365a-4129-898c-92c031497145" path="/var/lib/kubelet/pods/aaf50c8b-365a-4129-898c-92c031497145/volumes"
Sep 09 18:45:17.703606 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:17.703568    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-b4lhf"]
Sep 09 18:45:17.704119 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:17.703981    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c4473e-a8fc-4d37-9b3e-02c27738d636" containerName="authorino"
Sep 09 18:45:17.704119 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:17.703994    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c4473e-a8fc-4d37-9b3e-02c27738d636" containerName="authorino"
Sep 09 18:45:17.704119 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:17.704057    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c4473e-a8fc-4d37-9b3e-02c27738d636" containerName="authorino"
Sep 09 18:45:17.707067 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:17.707048    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-b4lhf"
Sep 09 18:45:17.709896 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:17.709865    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vs78j\""
Sep 09 18:45:17.710446 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:17.710424    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 18:45:17.726937 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:17.726900    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-b4lhf"]
Sep 09 18:45:17.797128 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:17.797086    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq9q2\" (UniqueName: \"kubernetes.io/projected/69804e5e-c502-42ae-aee8-d4e2f7ea2262-kube-api-access-zq9q2\") pod \"payload-processing-5b54b7fbb7-b4lhf\" (UID: \"69804e5e-c502-42ae-aee8-d4e2f7ea2262\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-b4lhf"
Sep 09 18:45:17.797310 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:17.797251    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69804e5e-c502-42ae-aee8-d4e2f7ea2262-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-b4lhf\" (UID: \"69804e5e-c502-42ae-aee8-d4e2f7ea2262\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-b4lhf"
Sep 09 18:45:17.898190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:17.898139    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69804e5e-c502-42ae-aee8-d4e2f7ea2262-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-b4lhf\" (UID: \"69804e5e-c502-42ae-aee8-d4e2f7ea2262\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-b4lhf"
Sep 09 18:45:17.898380 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:17.898213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq9q2\" (UniqueName: \"kubernetes.io/projected/69804e5e-c502-42ae-aee8-d4e2f7ea2262-kube-api-access-zq9q2\") pod \"payload-processing-5b54b7fbb7-b4lhf\" (UID: \"69804e5e-c502-42ae-aee8-d4e2f7ea2262\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-b4lhf"
Sep 09 18:45:17.899053 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:17.899021    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69804e5e-c502-42ae-aee8-d4e2f7ea2262-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-b4lhf\" (UID: \"69804e5e-c502-42ae-aee8-d4e2f7ea2262\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-b4lhf"
Sep 09 18:45:17.915509 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:17.915461    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq9q2\" (UniqueName: \"kubernetes.io/projected/69804e5e-c502-42ae-aee8-d4e2f7ea2262-kube-api-access-zq9q2\") pod \"payload-processing-5b54b7fbb7-b4lhf\" (UID: \"69804e5e-c502-42ae-aee8-d4e2f7ea2262\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-b4lhf"
Sep 09 18:45:18.019606 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:18.019482    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-b4lhf"
Sep 09 18:45:18.327287 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:18.327206    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-b4lhf"]
Sep 09 18:45:18.330838 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:45:18.330808    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69804e5e_c502_42ae_aee8_d4e2f7ea2262.slice/crio-afe70222839359a6b4c88818a044715c123fd3e39497b8df15c70f44ff9ccd6a WatchSource:0}: Error finding container afe70222839359a6b4c88818a044715c123fd3e39497b8df15c70f44ff9ccd6a: Status 404 returned error can't find the container with id afe70222839359a6b4c88818a044715c123fd3e39497b8df15c70f44ff9ccd6a
Sep 09 18:45:18.747377 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:18.747342    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-b4lhf" event={"ID":"69804e5e-c502-42ae-aee8-d4e2f7ea2262","Type":"ContainerStarted","Data":"afe70222839359a6b4c88818a044715c123fd3e39497b8df15c70f44ff9ccd6a"}
Sep 09 18:45:21.764057 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:21.764019    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-b4lhf" event={"ID":"69804e5e-c502-42ae-aee8-d4e2f7ea2262","Type":"ContainerStarted","Data":"8a0baaa3ee3423ef75c87edef81e7d1e9c86347660cc885e186c0c980ee27292"}
Sep 09 18:45:21.764529 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:21.764142    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-b4lhf"
Sep 09 18:45:21.786226 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:21.786162    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-b4lhf" podStartSLOduration=2.060464308 podStartE2EDuration="4.786139153s" totalImagesPullingTime="2.725674845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:45:21.785846651 +0000 UTC m=+810.023216466" watchObservedRunningTime="2026-09-09 18:45:21.786139153 +0000 UTC m=+810.023508968"
Sep 09 18:45:32.772689 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:32.772654    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-b4lhf"
Sep 09 18:45:32.952087 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:32.952048    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"]
Sep 09 18:45:32.956535 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:32.956490    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:32.959537 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:32.959494    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 18:45:32.959715 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:32.959511    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9cfsg\""
Sep 09 18:45:32.959715 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:32.959560    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 18:45:32.959715 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:32.959549    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 09 18:45:32.971273 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:32.971238    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"]
Sep 09 18:45:33.044643 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.044549    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9d584a4-1535-4fde-8387-850b2754b8c8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.044643 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.044604    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9d584a4-1535-4fde-8387-850b2754b8c8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.044643 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.044645    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9d584a4-1535-4fde-8387-850b2754b8c8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.044879 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.044668    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9d584a4-1535-4fde-8387-850b2754b8c8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.044879 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.044765    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5877v\" (UniqueName: \"kubernetes.io/projected/a9d584a4-1535-4fde-8387-850b2754b8c8-kube-api-access-5877v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.044879 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.044801    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9d584a4-1535-4fde-8387-850b2754b8c8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.145815 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.145777    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9d584a4-1535-4fde-8387-850b2754b8c8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.145995 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.145855    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5877v\" (UniqueName: \"kubernetes.io/projected/a9d584a4-1535-4fde-8387-850b2754b8c8-kube-api-access-5877v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.145995 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.145876    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9d584a4-1535-4fde-8387-850b2754b8c8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.145995 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.145910    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9d584a4-1535-4fde-8387-850b2754b8c8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.145995 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.145945    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9d584a4-1535-4fde-8387-850b2754b8c8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.145995 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.145974    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9d584a4-1535-4fde-8387-850b2754b8c8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.146398 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.146372    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9d584a4-1535-4fde-8387-850b2754b8c8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.146398 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.146386    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9d584a4-1535-4fde-8387-850b2754b8c8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.146531 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.146429    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9d584a4-1535-4fde-8387-850b2754b8c8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.148373 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.148342    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9d584a4-1535-4fde-8387-850b2754b8c8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.148625 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.148607    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9d584a4-1535-4fde-8387-850b2754b8c8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.156642 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.156612    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5877v\" (UniqueName: \"kubernetes.io/projected/a9d584a4-1535-4fde-8387-850b2754b8c8-kube-api-access-5877v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78\" (UID: \"a9d584a4-1535-4fde-8387-850b2754b8c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.268489 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.268435    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:33.562441 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.562410    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"]
Sep 09 18:45:33.564153 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:45:33.564119    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d584a4_1535_4fde_8387_850b2754b8c8.slice/crio-cd05fc07828ff4d4ac81c6d4aee575000f29f69e38c2908d148435e7448cb83a WatchSource:0}: Error finding container cd05fc07828ff4d4ac81c6d4aee575000f29f69e38c2908d148435e7448cb83a: Status 404 returned error can't find the container with id cd05fc07828ff4d4ac81c6d4aee575000f29f69e38c2908d148435e7448cb83a
Sep 09 18:45:33.837910 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:33.837858    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78" event={"ID":"a9d584a4-1535-4fde-8387-850b2754b8c8","Type":"ContainerStarted","Data":"cd05fc07828ff4d4ac81c6d4aee575000f29f69e38c2908d148435e7448cb83a"}
Sep 09 18:45:35.852574 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:35.852537    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78" event={"ID":"a9d584a4-1535-4fde-8387-850b2754b8c8","Type":"ContainerStarted","Data":"d5b8ae8c60cb8e24ef03f1bb60471f15457efc688602f834f9baac9b14e43280"}
Sep 09 18:45:35.872581 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:35.872525    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78" podStartSLOduration=2.624211472 podStartE2EDuration="3.872491929s" totalImagesPullingTime="1.248280457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:45:35.872078474 +0000 UTC m=+824.109448289" watchObservedRunningTime="2026-09-09 18:45:35.872491929 +0000 UTC m=+824.109861744"
Sep 09 18:45:43.268816 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:43.268774    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:43.269237 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:43.268888    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:43.281312 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:43.281274    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:45:43.903519 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:45:43.903473    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78"
Sep 09 18:46:21.851858 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:21.851818    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"]
Sep 09 18:46:21.854437 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:21.854419    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:21.856832 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:21.856802    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 18:46:21.865248 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:21.865216    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"]
Sep 09 18:46:21.908748 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:21.908700    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2184298b-d8f3-4c77-a654-7c383d714aff-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:21.908937 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:21.908770    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd744\" (UniqueName: \"kubernetes.io/projected/2184298b-d8f3-4c77-a654-7c383d714aff-kube-api-access-rd744\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:21.908937 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:21.908843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2184298b-d8f3-4c77-a654-7c383d714aff-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:21.908937 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:21.908876    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2184298b-d8f3-4c77-a654-7c383d714aff-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:21.908937 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:21.908918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2184298b-d8f3-4c77-a654-7c383d714aff-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:21.909071 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:21.908976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2184298b-d8f3-4c77-a654-7c383d714aff-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:22.009950 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:22.009908    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2184298b-d8f3-4c77-a654-7c383d714aff-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:22.009950 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:22.009950    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2184298b-d8f3-4c77-a654-7c383d714aff-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:22.010234 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:22.009982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2184298b-d8f3-4c77-a654-7c383d714aff-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:22.010234 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:22.010033    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2184298b-d8f3-4c77-a654-7c383d714aff-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:22.010234 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:22.010106    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2184298b-d8f3-4c77-a654-7c383d714aff-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:22.010234 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:22.010178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd744\" (UniqueName: \"kubernetes.io/projected/2184298b-d8f3-4c77-a654-7c383d714aff-kube-api-access-rd744\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:22.010512 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:22.010467    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2184298b-d8f3-4c77-a654-7c383d714aff-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:22.010512 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:22.010481    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2184298b-d8f3-4c77-a654-7c383d714aff-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:22.010640 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:22.010570    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2184298b-d8f3-4c77-a654-7c383d714aff-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:22.012313 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:22.012290    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2184298b-d8f3-4c77-a654-7c383d714aff-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:22.012581 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:22.012564    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2184298b-d8f3-4c77-a654-7c383d714aff-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:22.018021 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:22.017985    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd744\" (UniqueName: \"kubernetes.io/projected/2184298b-d8f3-4c77-a654-7c383d714aff-kube-api-access-rd744\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j\" (UID: \"2184298b-d8f3-4c77-a654-7c383d714aff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:22.165457 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:22.165420    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:22.439735 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:22.439707    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"]
Sep 09 18:46:22.442580 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:46:22.442491    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2184298b_d8f3_4c77_a654_7c383d714aff.slice/crio-2161f0710c06acce0819b2921b52ab209b74800ac159f2e15f2adc1748570299 WatchSource:0}: Error finding container 2161f0710c06acce0819b2921b52ab209b74800ac159f2e15f2adc1748570299: Status 404 returned error can't find the container with id 2161f0710c06acce0819b2921b52ab209b74800ac159f2e15f2adc1748570299
Sep 09 18:46:23.087137 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:23.087098    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j" event={"ID":"2184298b-d8f3-4c77-a654-7c383d714aff","Type":"ContainerStarted","Data":"f0d6d904034b6b4e09f05673d6f035cb72fbe2688da6059987ff135394f54954"}
Sep 09 18:46:23.087776 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:23.087149    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j" event={"ID":"2184298b-d8f3-4c77-a654-7c383d714aff","Type":"ContainerStarted","Data":"2161f0710c06acce0819b2921b52ab209b74800ac159f2e15f2adc1748570299"}
Sep 09 18:46:23.104767 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:23.104712    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j" podStartSLOduration=1.919479612 podStartE2EDuration="2.104694105s" totalImagesPullingTime="185.214493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:46:23.103729567 +0000 UTC m=+871.341099384" watchObservedRunningTime="2026-09-09 18:46:23.104694105 +0000 UTC m=+871.342063920"
Sep 09 18:46:32.166318 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:32.166285    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:32.166704 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:32.166356    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:32.179260 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:32.179223    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:33.146066 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:33.146035    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j"
Sep 09 18:46:48.945807 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:48.945764    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"]
Sep 09 18:46:48.948337 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:48.948317    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:48.950836 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:48.950813    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 18:46:48.962336 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:48.962304    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"]
Sep 09 18:46:49.079395 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.079350    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1e4710a-d531-4339-a84c-4e00d878523e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.079599 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.079407    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1e4710a-d531-4339-a84c-4e00d878523e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.079599 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.079451    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1e4710a-d531-4339-a84c-4e00d878523e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.079599 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.079473    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1e4710a-d531-4339-a84c-4e00d878523e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.079599 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.079577    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1e4710a-d531-4339-a84c-4e00d878523e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.079599 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.079597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtbmn\" (UniqueName: \"kubernetes.io/projected/d1e4710a-d531-4339-a84c-4e00d878523e-kube-api-access-rtbmn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.181020 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.180970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1e4710a-d531-4339-a84c-4e00d878523e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.181020 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.181022    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1e4710a-d531-4339-a84c-4e00d878523e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.181296 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.181077    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1e4710a-d531-4339-a84c-4e00d878523e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.181296 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.181107    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtbmn\" (UniqueName: \"kubernetes.io/projected/d1e4710a-d531-4339-a84c-4e00d878523e-kube-api-access-rtbmn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.181296 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.181161    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1e4710a-d531-4339-a84c-4e00d878523e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.181296 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.181213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1e4710a-d531-4339-a84c-4e00d878523e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.181563 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.181487    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1e4710a-d531-4339-a84c-4e00d878523e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.181563 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.181551    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1e4710a-d531-4339-a84c-4e00d878523e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.181680 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.181602    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1e4710a-d531-4339-a84c-4e00d878523e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.183569 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.183546    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1e4710a-d531-4339-a84c-4e00d878523e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.183835 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.183817    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1e4710a-d531-4339-a84c-4e00d878523e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.190391 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.190361    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtbmn\" (UniqueName: \"kubernetes.io/projected/d1e4710a-d531-4339-a84c-4e00d878523e-kube-api-access-rtbmn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz\" (UID: \"d1e4710a-d531-4339-a84c-4e00d878523e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.258401 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.258298    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:49.531579 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:49.531549    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"]
Sep 09 18:46:49.534231 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:46:49.534198    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1e4710a_d531_4339_a84c_4e00d878523e.slice/crio-69fe3bad4f617876e4d381b5db23540dbb35d05012d2d3359523319223bb0c7b WatchSource:0}: Error finding container 69fe3bad4f617876e4d381b5db23540dbb35d05012d2d3359523319223bb0c7b: Status 404 returned error can't find the container with id 69fe3bad4f617876e4d381b5db23540dbb35d05012d2d3359523319223bb0c7b
Sep 09 18:46:50.220862 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:50.220828    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz" event={"ID":"d1e4710a-d531-4339-a84c-4e00d878523e","Type":"ContainerStarted","Data":"95c793754f84d5b5a38fc5a10398729045e309fd7376424f0de1e1a157c85493"}
Sep 09 18:46:50.220862 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:50.220876    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz" event={"ID":"d1e4710a-d531-4339-a84c-4e00d878523e","Type":"ContainerStarted","Data":"69fe3bad4f617876e4d381b5db23540dbb35d05012d2d3359523319223bb0c7b"}
Sep 09 18:46:50.238859 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:50.238803    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz" podStartSLOduration=2.071845502 podStartE2EDuration="2.238783857s" totalImagesPullingTime="166.938355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:46:50.236870527 +0000 UTC m=+898.474240347" watchObservedRunningTime="2026-09-09 18:46:50.238783857 +0000 UTC m=+898.476153671"
Sep 09 18:46:52.261028 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:52.260997    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 18:46:52.263289 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:52.263253    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 18:46:59.259342 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:59.259288    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:59.259342 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:59.259356    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:46:59.272664 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:46:59.272627    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:47:00.280851 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:00.280817    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz"
Sep 09 18:47:49.070198 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:49.070155    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f6fc7bf65-lpkhr"]
Sep 09 18:47:49.072884 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:49.072864    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f6fc7bf65-lpkhr"
Sep 09 18:47:49.083138 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:49.083108    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f6fc7bf65-lpkhr"]
Sep 09 18:47:49.142469 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:49.142426    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0746fc4-4d76-49f3-9d34-627a0fd2cbaf-tls-cert\") pod \"authorino-f6fc7bf65-lpkhr\" (UID: \"b0746fc4-4d76-49f3-9d34-627a0fd2cbaf\") " pod="kuadrant-system/authorino-f6fc7bf65-lpkhr"
Sep 09 18:47:49.142681 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:49.142484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b856w\" (UniqueName: \"kubernetes.io/projected/b0746fc4-4d76-49f3-9d34-627a0fd2cbaf-kube-api-access-b856w\") pod \"authorino-f6fc7bf65-lpkhr\" (UID: \"b0746fc4-4d76-49f3-9d34-627a0fd2cbaf\") " pod="kuadrant-system/authorino-f6fc7bf65-lpkhr"
Sep 09 18:47:49.243597 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:49.243548    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b856w\" (UniqueName: \"kubernetes.io/projected/b0746fc4-4d76-49f3-9d34-627a0fd2cbaf-kube-api-access-b856w\") pod \"authorino-f6fc7bf65-lpkhr\" (UID: \"b0746fc4-4d76-49f3-9d34-627a0fd2cbaf\") " pod="kuadrant-system/authorino-f6fc7bf65-lpkhr"
Sep 09 18:47:49.243807 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:49.243713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0746fc4-4d76-49f3-9d34-627a0fd2cbaf-tls-cert\") pod \"authorino-f6fc7bf65-lpkhr\" (UID: \"b0746fc4-4d76-49f3-9d34-627a0fd2cbaf\") " pod="kuadrant-system/authorino-f6fc7bf65-lpkhr"
Sep 09 18:47:49.246198 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:49.246166    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b0746fc4-4d76-49f3-9d34-627a0fd2cbaf-tls-cert\") pod \"authorino-f6fc7bf65-lpkhr\" (UID: \"b0746fc4-4d76-49f3-9d34-627a0fd2cbaf\") " pod="kuadrant-system/authorino-f6fc7bf65-lpkhr"
Sep 09 18:47:49.252352 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:49.252320    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b856w\" (UniqueName: \"kubernetes.io/projected/b0746fc4-4d76-49f3-9d34-627a0fd2cbaf-kube-api-access-b856w\") pod \"authorino-f6fc7bf65-lpkhr\" (UID: \"b0746fc4-4d76-49f3-9d34-627a0fd2cbaf\") " pod="kuadrant-system/authorino-f6fc7bf65-lpkhr"
Sep 09 18:47:49.382790 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:49.382751    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f6fc7bf65-lpkhr"
Sep 09 18:47:49.655165 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:49.655105    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f6fc7bf65-lpkhr"]
Sep 09 18:47:49.657818 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:47:49.657782    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0746fc4_4d76_49f3_9d34_627a0fd2cbaf.slice/crio-bdfc302dc19fb1a4d847754ea3afe895c24ebead66e4dc8887113a178022c677 WatchSource:0}: Error finding container bdfc302dc19fb1a4d847754ea3afe895c24ebead66e4dc8887113a178022c677: Status 404 returned error can't find the container with id bdfc302dc19fb1a4d847754ea3afe895c24ebead66e4dc8887113a178022c677
Sep 09 18:47:50.526263 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:50.526219    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f6fc7bf65-lpkhr" event={"ID":"b0746fc4-4d76-49f3-9d34-627a0fd2cbaf","Type":"ContainerStarted","Data":"bdfc302dc19fb1a4d847754ea3afe895c24ebead66e4dc8887113a178022c677"}
Sep 09 18:47:51.535404 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:51.535356    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f6fc7bf65-lpkhr" event={"ID":"b0746fc4-4d76-49f3-9d34-627a0fd2cbaf","Type":"ContainerStarted","Data":"93b794724d6dce8cf60c700514e2ec96e5deb5ee5794a33c22b99d43079bf7ad"}
Sep 09 18:47:51.556135 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:51.556064    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f6fc7bf65-lpkhr" podStartSLOduration=1.422608737 podStartE2EDuration="2.556037517s" totalImagesPullingTime="1.13342878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:47:51.553198294 +0000 UTC m=+959.790568110" watchObservedRunningTime="2026-09-09 18:47:51.556037517 +0000 UTC m=+959.793407333"
Sep 09 18:47:51.588523 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:51.588082    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c8df6648-twlfz"]
Sep 09 18:47:51.588523 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:51.588345    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6c8df6648-twlfz" podUID="0553582f-36c0-4143-baf9-579ca80ab2b1" containerName="authorino" containerID="cri-o://e760b4d0e5cf6d914a5606e735f0f2e0d96b05ba44691cec132e954f89129813" gracePeriod=30
Sep 09 18:47:51.869978 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:51.869946    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c8df6648-twlfz"
Sep 09 18:47:51.970632 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:51.970585    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0553582f-36c0-4143-baf9-579ca80ab2b1-tls-cert\") pod \"0553582f-36c0-4143-baf9-579ca80ab2b1\" (UID: \"0553582f-36c0-4143-baf9-579ca80ab2b1\") "
Sep 09 18:47:51.970841 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:51.970649    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpm5j\" (UniqueName: \"kubernetes.io/projected/0553582f-36c0-4143-baf9-579ca80ab2b1-kube-api-access-hpm5j\") pod \"0553582f-36c0-4143-baf9-579ca80ab2b1\" (UID: \"0553582f-36c0-4143-baf9-579ca80ab2b1\") "
Sep 09 18:47:51.974373 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:51.974329    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0553582f-36c0-4143-baf9-579ca80ab2b1-kube-api-access-hpm5j" (OuterVolumeSpecName: "kube-api-access-hpm5j") pod "0553582f-36c0-4143-baf9-579ca80ab2b1" (UID: "0553582f-36c0-4143-baf9-579ca80ab2b1"). InnerVolumeSpecName "kube-api-access-hpm5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:47:51.983274 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:51.983238    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0553582f-36c0-4143-baf9-579ca80ab2b1-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0553582f-36c0-4143-baf9-579ca80ab2b1" (UID: "0553582f-36c0-4143-baf9-579ca80ab2b1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:47:52.071777 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:52.071731    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpm5j\" (UniqueName: \"kubernetes.io/projected/0553582f-36c0-4143-baf9-579ca80ab2b1-kube-api-access-hpm5j\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:47:52.071777 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:52.071777    2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0553582f-36c0-4143-baf9-579ca80ab2b1-tls-cert\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:47:52.542660 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:52.542617    2575 generic.go:356] "Generic (PLEG): container finished" podID="0553582f-36c0-4143-baf9-579ca80ab2b1" containerID="e760b4d0e5cf6d914a5606e735f0f2e0d96b05ba44691cec132e954f89129813" exitCode=0
Sep 09 18:47:52.543133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:52.542680    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c8df6648-twlfz"
Sep 09 18:47:52.543133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:52.542677    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c8df6648-twlfz" event={"ID":"0553582f-36c0-4143-baf9-579ca80ab2b1","Type":"ContainerDied","Data":"e760b4d0e5cf6d914a5606e735f0f2e0d96b05ba44691cec132e954f89129813"}
Sep 09 18:47:52.543133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:52.542825    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c8df6648-twlfz" event={"ID":"0553582f-36c0-4143-baf9-579ca80ab2b1","Type":"ContainerDied","Data":"eab6a07532f3f216374dee7a55ce43d9b3c2873502bf5cd09db7603193426d4b"}
Sep 09 18:47:52.543133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:52.542866    2575 scope.go:117] "RemoveContainer" containerID="e760b4d0e5cf6d914a5606e735f0f2e0d96b05ba44691cec132e954f89129813"
Sep 09 18:47:52.552862 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:52.552839    2575 scope.go:117] "RemoveContainer" containerID="e760b4d0e5cf6d914a5606e735f0f2e0d96b05ba44691cec132e954f89129813"
Sep 09 18:47:52.553259 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:47:52.553228    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e760b4d0e5cf6d914a5606e735f0f2e0d96b05ba44691cec132e954f89129813\": container with ID starting with e760b4d0e5cf6d914a5606e735f0f2e0d96b05ba44691cec132e954f89129813 not found: ID does not exist" containerID="e760b4d0e5cf6d914a5606e735f0f2e0d96b05ba44691cec132e954f89129813"
Sep 09 18:47:52.553374 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:52.553268    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e760b4d0e5cf6d914a5606e735f0f2e0d96b05ba44691cec132e954f89129813"} err="failed to get container status \"e760b4d0e5cf6d914a5606e735f0f2e0d96b05ba44691cec132e954f89129813\": rpc error: code = NotFound desc = could not find container \"e760b4d0e5cf6d914a5606e735f0f2e0d96b05ba44691cec132e954f89129813\": container with ID starting with e760b4d0e5cf6d914a5606e735f0f2e0d96b05ba44691cec132e954f89129813 not found: ID does not exist"
Sep 09 18:47:52.562672 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:52.562640    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c8df6648-twlfz"]
Sep 09 18:47:52.565749 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:52.565720    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c8df6648-twlfz"]
Sep 09 18:47:54.321083 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:47:54.321043    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0553582f-36c0-4143-baf9-579ca80ab2b1" path="/var/lib/kubelet/pods/0553582f-36c0-4143-baf9-579ca80ab2b1/volumes"
Sep 09 18:48:14.979400 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:14.979317    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"]
Sep 09 18:48:14.979907 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:14.979745    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0553582f-36c0-4143-baf9-579ca80ab2b1" containerName="authorino"
Sep 09 18:48:14.979907 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:14.979759    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0553582f-36c0-4143-baf9-579ca80ab2b1" containerName="authorino"
Sep 09 18:48:14.979907 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:14.979833    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0553582f-36c0-4143-baf9-579ca80ab2b1" containerName="authorino"
Sep 09 18:48:14.989562 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:14.989528    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"
Sep 09 18:48:14.992630 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:14.992600    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-90f815\""
Sep 09 18:48:14.992807 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:14.992600    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-90f815-dockercfg-x686q\""
Sep 09 18:48:14.994280 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:14.994252    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"]
Sep 09 18:48:15.096801 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:15.096762    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2n7s\" (UniqueName: \"kubernetes.io/projected/3d49c131-9c66-4c5f-abd1-2afe90aeab15-kube-api-access-m2n7s\") pod \"payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8\" (UID: \"3d49c131-9c66-4c5f-abd1-2afe90aeab15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"
Sep 09 18:48:15.096985 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:15.096812    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d49c131-9c66-4c5f-abd1-2afe90aeab15-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8\" (UID: \"3d49c131-9c66-4c5f-abd1-2afe90aeab15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"
Sep 09 18:48:15.197952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:15.197913    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d49c131-9c66-4c5f-abd1-2afe90aeab15-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8\" (UID: \"3d49c131-9c66-4c5f-abd1-2afe90aeab15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"
Sep 09 18:48:15.198141 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:15.198052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2n7s\" (UniqueName: \"kubernetes.io/projected/3d49c131-9c66-4c5f-abd1-2afe90aeab15-kube-api-access-m2n7s\") pod \"payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8\" (UID: \"3d49c131-9c66-4c5f-abd1-2afe90aeab15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"
Sep 09 18:48:15.198684 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:15.198662    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d49c131-9c66-4c5f-abd1-2afe90aeab15-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8\" (UID: \"3d49c131-9c66-4c5f-abd1-2afe90aeab15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"
Sep 09 18:48:15.206483 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:15.206456    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2n7s\" (UniqueName: \"kubernetes.io/projected/3d49c131-9c66-4c5f-abd1-2afe90aeab15-kube-api-access-m2n7s\") pod \"payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8\" (UID: \"3d49c131-9c66-4c5f-abd1-2afe90aeab15\") " pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"
Sep 09 18:48:15.301725 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:15.301624    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"
Sep 09 18:48:15.576019 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:15.575898    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"]
Sep 09 18:48:15.579001 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:48:15.578965    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d49c131_9c66_4c5f_abd1_2afe90aeab15.slice/crio-017c657b5f590d30e4f4c61b35730196ff653f0c6785b244af1a161a596d78b4 WatchSource:0}: Error finding container 017c657b5f590d30e4f4c61b35730196ff653f0c6785b244af1a161a596d78b4: Status 404 returned error can't find the container with id 017c657b5f590d30e4f4c61b35730196ff653f0c6785b244af1a161a596d78b4
Sep 09 18:48:15.663163 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:15.663125    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8" event={"ID":"3d49c131-9c66-4c5f-abd1-2afe90aeab15","Type":"ContainerStarted","Data":"f59171de851f3e11bfbd87df26cd227d7a5aa0d829789059e218204a30ba5123"}
Sep 09 18:48:15.663380 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:15.663194    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"
Sep 09 18:48:15.663380 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:15.663214    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8" event={"ID":"3d49c131-9c66-4c5f-abd1-2afe90aeab15","Type":"ContainerStarted","Data":"017c657b5f590d30e4f4c61b35730196ff653f0c6785b244af1a161a596d78b4"}
Sep 09 18:48:15.680746 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:15.680691    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8" podStartSLOduration=1.680671259 podStartE2EDuration="1.680671259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:48:15.679683542 +0000 UTC m=+983.917053387" watchObservedRunningTime="2026-09-09 18:48:15.680671259 +0000 UTC m=+983.918041076"
Sep 09 18:48:17.329733 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.329694    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"]
Sep 09 18:48:17.332657 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.332631    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.335236 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.335211    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-8b6cce27-kserve-self-signed-certs\""
Sep 09 18:48:17.344089 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.344058    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"]
Sep 09 18:48:17.420360 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.420311    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-model-cache\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.420360 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.420363    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxfwf\" (UniqueName: \"kubernetes.io/projected/e5f0450b-b600-45c5-afc8-70d24d65d905-kube-api-access-bxfwf\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.420675 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.420442    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-home\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.420675 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.420490    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-dshm\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.420675 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.420550    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-tmp-dir\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.420675 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.420584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f0450b-b600-45c5-afc8-70d24d65d905-tls-certs\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.522088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.522031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-model-cache\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.522268 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.522192    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxfwf\" (UniqueName: \"kubernetes.io/projected/e5f0450b-b600-45c5-afc8-70d24d65d905-kube-api-access-bxfwf\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.522268 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.522244    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-home\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.522352 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.522277    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-dshm\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.522352 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.522304    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-tmp-dir\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.522352 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.522341    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f0450b-b600-45c5-afc8-70d24d65d905-tls-certs\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.522521 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.522472    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-model-cache\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.522646 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.522627    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-home\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.522712 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.522693    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-tmp-dir\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.524774 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.524747    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-dshm\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.525024 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.525005    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f0450b-b600-45c5-afc8-70d24d65d905-tls-certs\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.533410 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.533385    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxfwf\" (UniqueName: \"kubernetes.io/projected/e5f0450b-b600-45c5-afc8-70d24d65d905-kube-api-access-bxfwf\") pod \"e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") " pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.642452 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.642412    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:17.930269 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:17.930243    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"]
Sep 09 18:48:17.932047 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:48:17.932016    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5f0450b_b600_45c5_afc8_70d24d65d905.slice/crio-894abc4305ba2224672338655834fa180b5450acce90dd5f77290719aee245c4 WatchSource:0}: Error finding container 894abc4305ba2224672338655834fa180b5450acce90dd5f77290719aee245c4: Status 404 returned error can't find the container with id 894abc4305ba2224672338655834fa180b5450acce90dd5f77290719aee245c4
Sep 09 18:48:18.685342 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:18.685289    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7" event={"ID":"e5f0450b-b600-45c5-afc8-70d24d65d905","Type":"ContainerStarted","Data":"1642cea8b719ce357b12372b2959d1c22c96c26b612af8f6718d136cc577d511"}
Sep 09 18:48:18.685342 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:18.685347    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7" event={"ID":"e5f0450b-b600-45c5-afc8-70d24d65d905","Type":"ContainerStarted","Data":"894abc4305ba2224672338655834fa180b5450acce90dd5f77290719aee245c4"}
Sep 09 18:48:18.704116 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:18.704048    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7" podStartSLOduration=1.704023906 podStartE2EDuration="1.704023906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:48:18.702271567 +0000 UTC m=+986.939641408" watchObservedRunningTime="2026-09-09 18:48:18.704023906 +0000 UTC m=+986.941393725"
Sep 09 18:48:22.475682 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.475640    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"]
Sep 09 18:48:22.478733 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.478706    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"
Sep 09 18:48:22.482665 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.482636    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d0c2fa5c\""
Sep 09 18:48:22.482830 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.482632    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d0c2fa5c-dockercfg-v9cj6\""
Sep 09 18:48:22.499942 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.499908    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"]
Sep 09 18:48:22.532651 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.532617    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"]
Sep 09 18:48:22.535274 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.535254    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"
Sep 09 18:48:22.552470 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.552435    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"]
Sep 09 18:48:22.573459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.573420    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ad5922-1a81-4600-bc40-a409d1204c9c-plugins-config-volume\") pod \"payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps\" (UID: \"40ad5922-1a81-4600-bc40-a409d1204c9c\") " pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"
Sep 09 18:48:22.573687 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.573551    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2krxz\" (UniqueName: \"kubernetes.io/projected/15198d8d-3540-4e9e-b4d3-de94a67aa733-kube-api-access-2krxz\") pod \"payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks\" (UID: \"15198d8d-3540-4e9e-b4d3-de94a67aa733\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"
Sep 09 18:48:22.573687 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.573609    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pkrz\" (UniqueName: \"kubernetes.io/projected/40ad5922-1a81-4600-bc40-a409d1204c9c-kube-api-access-5pkrz\") pod \"payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps\" (UID: \"40ad5922-1a81-4600-bc40-a409d1204c9c\") " pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"
Sep 09 18:48:22.573687 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.573657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15198d8d-3540-4e9e-b4d3-de94a67aa733-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks\" (UID: \"15198d8d-3540-4e9e-b4d3-de94a67aa733\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"
Sep 09 18:48:22.674708 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.674663    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2krxz\" (UniqueName: \"kubernetes.io/projected/15198d8d-3540-4e9e-b4d3-de94a67aa733-kube-api-access-2krxz\") pod \"payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks\" (UID: \"15198d8d-3540-4e9e-b4d3-de94a67aa733\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"
Sep 09 18:48:22.674915 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.674719    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pkrz\" (UniqueName: \"kubernetes.io/projected/40ad5922-1a81-4600-bc40-a409d1204c9c-kube-api-access-5pkrz\") pod \"payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps\" (UID: \"40ad5922-1a81-4600-bc40-a409d1204c9c\") " pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"
Sep 09 18:48:22.674915 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.674755    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15198d8d-3540-4e9e-b4d3-de94a67aa733-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks\" (UID: \"15198d8d-3540-4e9e-b4d3-de94a67aa733\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"
Sep 09 18:48:22.674915 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.674835    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ad5922-1a81-4600-bc40-a409d1204c9c-plugins-config-volume\") pod \"payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps\" (UID: \"40ad5922-1a81-4600-bc40-a409d1204c9c\") " pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"
Sep 09 18:48:22.675639 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.675611    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15198d8d-3540-4e9e-b4d3-de94a67aa733-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks\" (UID: \"15198d8d-3540-4e9e-b4d3-de94a67aa733\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"
Sep 09 18:48:22.675753 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.675618    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ad5922-1a81-4600-bc40-a409d1204c9c-plugins-config-volume\") pod \"payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps\" (UID: \"40ad5922-1a81-4600-bc40-a409d1204c9c\") " pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"
Sep 09 18:48:22.682615 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.682581    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2krxz\" (UniqueName: \"kubernetes.io/projected/15198d8d-3540-4e9e-b4d3-de94a67aa733-kube-api-access-2krxz\") pod \"payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks\" (UID: \"15198d8d-3540-4e9e-b4d3-de94a67aa733\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"
Sep 09 18:48:22.682615 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.682602    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pkrz\" (UniqueName: \"kubernetes.io/projected/40ad5922-1a81-4600-bc40-a409d1204c9c-kube-api-access-5pkrz\") pod \"payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps\" (UID: \"40ad5922-1a81-4600-bc40-a409d1204c9c\") " pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"
Sep 09 18:48:22.790934 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.790829    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"
Sep 09 18:48:22.847993 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:22.847956    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"
Sep 09 18:48:23.069247 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:23.069215    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"]
Sep 09 18:48:23.071294 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:48:23.071262    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15198d8d_3540_4e9e_b4d3_de94a67aa733.slice/crio-8394a776bf9fc9a77749c2c87a00078d4714b403f7d26c34264a9e6dd069e316 WatchSource:0}: Error finding container 8394a776bf9fc9a77749c2c87a00078d4714b403f7d26c34264a9e6dd069e316: Status 404 returned error can't find the container with id 8394a776bf9fc9a77749c2c87a00078d4714b403f7d26c34264a9e6dd069e316
Sep 09 18:48:23.134361 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:23.134335    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"]
Sep 09 18:48:23.136921 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:48:23.136895    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ad5922_1a81_4600_bc40_a409d1204c9c.slice/crio-280ac2c787c8c3c496460fcd5e14869100837e50a87d9c600318a4eaa15d3a07 WatchSource:0}: Error finding container 280ac2c787c8c3c496460fcd5e14869100837e50a87d9c600318a4eaa15d3a07: Status 404 returned error can't find the container with id 280ac2c787c8c3c496460fcd5e14869100837e50a87d9c600318a4eaa15d3a07
Sep 09 18:48:23.719200 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:23.719155    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks" event={"ID":"15198d8d-3540-4e9e-b4d3-de94a67aa733","Type":"ContainerStarted","Data":"b75535d7f17112dcee2e8d4de3b2e0b5ecd2cc9b022a3a9228280aa6ce47d2b2"}
Sep 09 18:48:23.719719 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:23.719344    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"
Sep 09 18:48:23.719719 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:23.719385    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks" event={"ID":"15198d8d-3540-4e9e-b4d3-de94a67aa733","Type":"ContainerStarted","Data":"8394a776bf9fc9a77749c2c87a00078d4714b403f7d26c34264a9e6dd069e316"}
Sep 09 18:48:23.720906 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:23.720883    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps" event={"ID":"40ad5922-1a81-4600-bc40-a409d1204c9c","Type":"ContainerStarted","Data":"de8c12244766fd4a36ebe61ea9771eb1202879bfa16e8ba58543e08444050c36"}
Sep 09 18:48:23.721003 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:23.720934    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps" event={"ID":"40ad5922-1a81-4600-bc40-a409d1204c9c","Type":"ContainerStarted","Data":"280ac2c787c8c3c496460fcd5e14869100837e50a87d9c600318a4eaa15d3a07"}
Sep 09 18:48:23.721003 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:23.720964    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"
Sep 09 18:48:23.738016 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:23.737939    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks" podStartSLOduration=1.737919193 podStartE2EDuration="1.737919193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:48:23.735439299 +0000 UTC m=+991.972809114" watchObservedRunningTime="2026-09-09 18:48:23.737919193 +0000 UTC m=+991.975289075"
Sep 09 18:48:23.756449 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:23.756387    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps" podStartSLOduration=1.756370582 podStartE2EDuration="1.756370582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:48:23.754211371 +0000 UTC m=+991.991581196" watchObservedRunningTime="2026-09-09 18:48:23.756370582 +0000 UTC m=+991.993740397"
Sep 09 18:48:26.671711 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:26.671677    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"
Sep 09 18:48:27.643156 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:27.643090    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:27.643156 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:27.643161    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:27.656457 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:27.656416    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:27.753977 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:27.753943    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:48:34.728100 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:34.728065    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"
Sep 09 18:48:34.728594 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:34.728145    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"
Sep 09 18:48:35.151209 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:35.151167    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"]
Sep 09 18:48:35.155111 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:35.155088    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"
Sep 09 18:48:35.158346 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:35.158321    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-38af9d-dockercfg-88hwz\""
Sep 09 18:48:35.158606 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:35.158589    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-38af9d\""
Sep 09 18:48:35.167229 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:35.167199    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"]
Sep 09 18:48:35.200100 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:35.200068    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6799af4e-e6fc-43bc-9423-082e9429ae9f-plugins-config-volume\") pod \"payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq\" (UID: \"6799af4e-e6fc-43bc-9423-082e9429ae9f\") " pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"
Sep 09 18:48:35.200100 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:35.200109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk5v2\" (UniqueName: \"kubernetes.io/projected/6799af4e-e6fc-43bc-9423-082e9429ae9f-kube-api-access-xk5v2\") pod \"payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq\" (UID: \"6799af4e-e6fc-43bc-9423-082e9429ae9f\") " pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"
Sep 09 18:48:35.301343 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:35.301294    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6799af4e-e6fc-43bc-9423-082e9429ae9f-plugins-config-volume\") pod \"payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq\" (UID: \"6799af4e-e6fc-43bc-9423-082e9429ae9f\") " pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"
Sep 09 18:48:35.301343 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:35.301346    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk5v2\" (UniqueName: \"kubernetes.io/projected/6799af4e-e6fc-43bc-9423-082e9429ae9f-kube-api-access-xk5v2\") pod \"payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq\" (UID: \"6799af4e-e6fc-43bc-9423-082e9429ae9f\") " pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"
Sep 09 18:48:35.302152 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:35.302129    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6799af4e-e6fc-43bc-9423-082e9429ae9f-plugins-config-volume\") pod \"payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq\" (UID: \"6799af4e-e6fc-43bc-9423-082e9429ae9f\") " pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"
Sep 09 18:48:35.310326 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:35.310293    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk5v2\" (UniqueName: \"kubernetes.io/projected/6799af4e-e6fc-43bc-9423-082e9429ae9f-kube-api-access-xk5v2\") pod \"payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq\" (UID: \"6799af4e-e6fc-43bc-9423-082e9429ae9f\") " pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"
Sep 09 18:48:35.467926 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:35.467827    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"
Sep 09 18:48:35.745434 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:35.745401    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"]
Sep 09 18:48:35.746969 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:48:35.746938    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6799af4e_e6fc_43bc_9423_082e9429ae9f.slice/crio-4e09ac0928225bab8b4d5d3ccede0a4d41d539b7893dc5c81c68091d58e7196d WatchSource:0}: Error finding container 4e09ac0928225bab8b4d5d3ccede0a4d41d539b7893dc5c81c68091d58e7196d: Status 404 returned error can't find the container with id 4e09ac0928225bab8b4d5d3ccede0a4d41d539b7893dc5c81c68091d58e7196d
Sep 09 18:48:35.785629 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:35.785601    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq" event={"ID":"6799af4e-e6fc-43bc-9423-082e9429ae9f","Type":"ContainerStarted","Data":"4e09ac0928225bab8b4d5d3ccede0a4d41d539b7893dc5c81c68091d58e7196d"}
Sep 09 18:48:36.793658 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:36.793610    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq" event={"ID":"6799af4e-e6fc-43bc-9423-082e9429ae9f","Type":"ContainerStarted","Data":"b5b173fa819578769e1ce55091b3d2900e9e8cf6f474c7331f63f9521006cd63"}
Sep 09 18:48:36.794078 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:36.793758    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"
Sep 09 18:48:36.814116 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:36.814051    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq" podStartSLOduration=1.8140316680000002 podStartE2EDuration="1.814031668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:48:36.81229202 +0000 UTC m=+1005.049661835" watchObservedRunningTime="2026-09-09 18:48:36.814031668 +0000 UTC m=+1005.051401482"
Sep 09 18:48:43.272078 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.272039    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"]
Sep 09 18:48:43.274926 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.274907    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"
Sep 09 18:48:43.281444 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.281416    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-77aadf\""
Sep 09 18:48:43.297121 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.297083    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"]
Sep 09 18:48:43.300528 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.300486    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"]
Sep 09 18:48:43.300686 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.300608    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"
Sep 09 18:48:43.304658 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.304629    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-77aadf-dockercfg-wxbgc\""
Sep 09 18:48:43.314210 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.314178    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"]
Sep 09 18:48:43.390599 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.390559    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-794qs\" (UniqueName: \"kubernetes.io/projected/627dada0-b2d6-4669-b7aa-183b47ea2539-kube-api-access-794qs\") pod \"payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd\" (UID: \"627dada0-b2d6-4669-b7aa-183b47ea2539\") " pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"
Sep 09 18:48:43.390794 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.390692    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2d9l\" (UniqueName: \"kubernetes.io/projected/b4317af6-a191-4e52-b714-867c12233c03-kube-api-access-x2d9l\") pod \"payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh\" (UID: \"b4317af6-a191-4e52-b714-867c12233c03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"
Sep 09 18:48:43.390794 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.390766    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/627dada0-b2d6-4669-b7aa-183b47ea2539-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd\" (UID: \"627dada0-b2d6-4669-b7aa-183b47ea2539\") " pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"
Sep 09 18:48:43.390877 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.390818    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4317af6-a191-4e52-b714-867c12233c03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh\" (UID: \"b4317af6-a191-4e52-b714-867c12233c03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"
Sep 09 18:48:43.491919 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.491877    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/627dada0-b2d6-4669-b7aa-183b47ea2539-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd\" (UID: \"627dada0-b2d6-4669-b7aa-183b47ea2539\") " pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"
Sep 09 18:48:43.492138 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.491930    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4317af6-a191-4e52-b714-867c12233c03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh\" (UID: \"b4317af6-a191-4e52-b714-867c12233c03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"
Sep 09 18:48:43.492138 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.491962    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-794qs\" (UniqueName: \"kubernetes.io/projected/627dada0-b2d6-4669-b7aa-183b47ea2539-kube-api-access-794qs\") pod \"payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd\" (UID: \"627dada0-b2d6-4669-b7aa-183b47ea2539\") " pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"
Sep 09 18:48:43.492138 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.492028    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2d9l\" (UniqueName: \"kubernetes.io/projected/b4317af6-a191-4e52-b714-867c12233c03-kube-api-access-x2d9l\") pod \"payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh\" (UID: \"b4317af6-a191-4e52-b714-867c12233c03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"
Sep 09 18:48:43.492709 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.492686    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/627dada0-b2d6-4669-b7aa-183b47ea2539-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd\" (UID: \"627dada0-b2d6-4669-b7aa-183b47ea2539\") " pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"
Sep 09 18:48:43.492789 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.492768    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4317af6-a191-4e52-b714-867c12233c03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh\" (UID: \"b4317af6-a191-4e52-b714-867c12233c03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"
Sep 09 18:48:43.502131 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.502085    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-794qs\" (UniqueName: \"kubernetes.io/projected/627dada0-b2d6-4669-b7aa-183b47ea2539-kube-api-access-794qs\") pod \"payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd\" (UID: \"627dada0-b2d6-4669-b7aa-183b47ea2539\") " pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"
Sep 09 18:48:43.502518 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.502474    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2d9l\" (UniqueName: \"kubernetes.io/projected/b4317af6-a191-4e52-b714-867c12233c03-kube-api-access-x2d9l\") pod \"payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh\" (UID: \"b4317af6-a191-4e52-b714-867c12233c03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"
Sep 09 18:48:43.587272 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.587176    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"
Sep 09 18:48:43.613048 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.613016    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"
Sep 09 18:48:43.868158 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.868127    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"]
Sep 09 18:48:43.869829 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:48:43.869798    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4317af6_a191_4e52_b714_867c12233c03.slice/crio-8ca8f6ae850b8e558a284b73156dd5202a3f250207dea38a9cafb103a183ea47 WatchSource:0}: Error finding container 8ca8f6ae850b8e558a284b73156dd5202a3f250207dea38a9cafb103a183ea47: Status 404 returned error can't find the container with id 8ca8f6ae850b8e558a284b73156dd5202a3f250207dea38a9cafb103a183ea47
Sep 09 18:48:43.931627 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:43.931602    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"]
Sep 09 18:48:43.933712 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:48:43.933686    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627dada0_b2d6_4669_b7aa_183b47ea2539.slice/crio-86f16b29fdd65195fae81608c24435244aa6f7b281ce113480813f3a1083c932 WatchSource:0}: Error finding container 86f16b29fdd65195fae81608c24435244aa6f7b281ce113480813f3a1083c932: Status 404 returned error can't find the container with id 86f16b29fdd65195fae81608c24435244aa6f7b281ce113480813f3a1083c932
Sep 09 18:48:44.837092 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:44.837057    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd" event={"ID":"627dada0-b2d6-4669-b7aa-183b47ea2539","Type":"ContainerStarted","Data":"dc35da29a025ebe575b4ff686a30808a25572b77447128ec2c0b726c914131e6"}
Sep 09 18:48:44.837582 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:44.837118    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"
Sep 09 18:48:44.837582 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:44.837143    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd" event={"ID":"627dada0-b2d6-4669-b7aa-183b47ea2539","Type":"ContainerStarted","Data":"86f16b29fdd65195fae81608c24435244aa6f7b281ce113480813f3a1083c932"}
Sep 09 18:48:44.838751 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:44.838727    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh" event={"ID":"b4317af6-a191-4e52-b714-867c12233c03","Type":"ContainerStarted","Data":"6d5e54bb2778fb6a1c33da45a39190ca1555e468dd71c0f9d0e2460e5423f928"}
Sep 09 18:48:44.838845 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:44.838770    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh" event={"ID":"b4317af6-a191-4e52-b714-867c12233c03","Type":"ContainerStarted","Data":"8ca8f6ae850b8e558a284b73156dd5202a3f250207dea38a9cafb103a183ea47"}
Sep 09 18:48:44.838887 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:44.838855    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"
Sep 09 18:48:44.855432 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:44.855379    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd" podStartSLOduration=1.8553621470000001 podStartE2EDuration="1.855362147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:48:44.853985592 +0000 UTC m=+1013.091355408" watchObservedRunningTime="2026-09-09 18:48:44.855362147 +0000 UTC m=+1013.092731961"
Sep 09 18:48:44.876534 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:44.876444    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh" podStartSLOduration=1.876425641 podStartE2EDuration="1.876425641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:48:44.875191953 +0000 UTC m=+1013.112561770" watchObservedRunningTime="2026-09-09 18:48:44.876425641 +0000 UTC m=+1013.113795456"
Sep 09 18:48:47.801910 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:47.801877    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"
Sep 09 18:48:55.847540 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:55.847491    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"
Sep 09 18:48:55.848017 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:48:55.847587    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"
Sep 09 18:49:02.622395 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.622352    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"]
Sep 09 18:49:02.622949 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.622702    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps" podUID="40ad5922-1a81-4600-bc40-a409d1204c9c" containerName="payload-processing" containerID="cri-o://de8c12244766fd4a36ebe61ea9771eb1202879bfa16e8ba58543e08444050c36" gracePeriod=30
Sep 09 18:49:02.631659 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.631625    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"]
Sep 09 18:49:02.631960 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.631905    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks" podUID="15198d8d-3540-4e9e-b4d3-de94a67aa733" containerName="payload-pre-processing" containerID="cri-o://b75535d7f17112dcee2e8d4de3b2e0b5ecd2cc9b022a3a9228280aa6ce47d2b2" gracePeriod=30
Sep 09 18:49:02.885588 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.885515    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"
Sep 09 18:49:02.917936 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.917907    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"
Sep 09 18:49:02.950250 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.950218    2575 generic.go:356] "Generic (PLEG): container finished" podID="15198d8d-3540-4e9e-b4d3-de94a67aa733" containerID="b75535d7f17112dcee2e8d4de3b2e0b5ecd2cc9b022a3a9228280aa6ce47d2b2" exitCode=0
Sep 09 18:49:02.950436 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.950326    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"
Sep 09 18:49:02.950436 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.950322    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks" event={"ID":"15198d8d-3540-4e9e-b4d3-de94a67aa733","Type":"ContainerDied","Data":"b75535d7f17112dcee2e8d4de3b2e0b5ecd2cc9b022a3a9228280aa6ce47d2b2"}
Sep 09 18:49:02.950436 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.950388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks" event={"ID":"15198d8d-3540-4e9e-b4d3-de94a67aa733","Type":"ContainerDied","Data":"8394a776bf9fc9a77749c2c87a00078d4714b403f7d26c34264a9e6dd069e316"}
Sep 09 18:49:02.950436 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.950418    2575 scope.go:117] "RemoveContainer" containerID="b75535d7f17112dcee2e8d4de3b2e0b5ecd2cc9b022a3a9228280aa6ce47d2b2"
Sep 09 18:49:02.951973 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.951949    2575 generic.go:356] "Generic (PLEG): container finished" podID="40ad5922-1a81-4600-bc40-a409d1204c9c" containerID="de8c12244766fd4a36ebe61ea9771eb1202879bfa16e8ba58543e08444050c36" exitCode=0
Sep 09 18:49:02.952114 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.952004    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"
Sep 09 18:49:02.952114 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.952013    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps" event={"ID":"40ad5922-1a81-4600-bc40-a409d1204c9c","Type":"ContainerDied","Data":"de8c12244766fd4a36ebe61ea9771eb1202879bfa16e8ba58543e08444050c36"}
Sep 09 18:49:02.952114 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.952059    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps" event={"ID":"40ad5922-1a81-4600-bc40-a409d1204c9c","Type":"ContainerDied","Data":"280ac2c787c8c3c496460fcd5e14869100837e50a87d9c600318a4eaa15d3a07"}
Sep 09 18:49:02.960972 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.960948    2575 scope.go:117] "RemoveContainer" containerID="b75535d7f17112dcee2e8d4de3b2e0b5ecd2cc9b022a3a9228280aa6ce47d2b2"
Sep 09 18:49:02.961377 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:49:02.961351    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75535d7f17112dcee2e8d4de3b2e0b5ecd2cc9b022a3a9228280aa6ce47d2b2\": container with ID starting with b75535d7f17112dcee2e8d4de3b2e0b5ecd2cc9b022a3a9228280aa6ce47d2b2 not found: ID does not exist" containerID="b75535d7f17112dcee2e8d4de3b2e0b5ecd2cc9b022a3a9228280aa6ce47d2b2"
Sep 09 18:49:02.961734 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.961388    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75535d7f17112dcee2e8d4de3b2e0b5ecd2cc9b022a3a9228280aa6ce47d2b2"} err="failed to get container status \"b75535d7f17112dcee2e8d4de3b2e0b5ecd2cc9b022a3a9228280aa6ce47d2b2\": rpc error: code = NotFound desc = could not find container \"b75535d7f17112dcee2e8d4de3b2e0b5ecd2cc9b022a3a9228280aa6ce47d2b2\": container with ID starting with b75535d7f17112dcee2e8d4de3b2e0b5ecd2cc9b022a3a9228280aa6ce47d2b2 not found: ID does not exist"
Sep 09 18:49:02.961734 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.961408    2575 scope.go:117] "RemoveContainer" containerID="de8c12244766fd4a36ebe61ea9771eb1202879bfa16e8ba58543e08444050c36"
Sep 09 18:49:02.972533 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.972491    2575 scope.go:117] "RemoveContainer" containerID="de8c12244766fd4a36ebe61ea9771eb1202879bfa16e8ba58543e08444050c36"
Sep 09 18:49:02.972932 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:49:02.972899    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8c12244766fd4a36ebe61ea9771eb1202879bfa16e8ba58543e08444050c36\": container with ID starting with de8c12244766fd4a36ebe61ea9771eb1202879bfa16e8ba58543e08444050c36 not found: ID does not exist" containerID="de8c12244766fd4a36ebe61ea9771eb1202879bfa16e8ba58543e08444050c36"
Sep 09 18:49:02.973043 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.972943    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8c12244766fd4a36ebe61ea9771eb1202879bfa16e8ba58543e08444050c36"} err="failed to get container status \"de8c12244766fd4a36ebe61ea9771eb1202879bfa16e8ba58543e08444050c36\": rpc error: code = NotFound desc = could not find container \"de8c12244766fd4a36ebe61ea9771eb1202879bfa16e8ba58543e08444050c36\": container with ID starting with de8c12244766fd4a36ebe61ea9771eb1202879bfa16e8ba58543e08444050c36 not found: ID does not exist"
Sep 09 18:49:02.990307 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.990268    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ad5922-1a81-4600-bc40-a409d1204c9c-plugins-config-volume\") pod \"40ad5922-1a81-4600-bc40-a409d1204c9c\" (UID: \"40ad5922-1a81-4600-bc40-a409d1204c9c\") "
Sep 09 18:49:02.990307 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.990312    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2krxz\" (UniqueName: \"kubernetes.io/projected/15198d8d-3540-4e9e-b4d3-de94a67aa733-kube-api-access-2krxz\") pod \"15198d8d-3540-4e9e-b4d3-de94a67aa733\" (UID: \"15198d8d-3540-4e9e-b4d3-de94a67aa733\") "
Sep 09 18:49:02.990593 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.990413    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pkrz\" (UniqueName: \"kubernetes.io/projected/40ad5922-1a81-4600-bc40-a409d1204c9c-kube-api-access-5pkrz\") pod \"40ad5922-1a81-4600-bc40-a409d1204c9c\" (UID: \"40ad5922-1a81-4600-bc40-a409d1204c9c\") "
Sep 09 18:49:02.990593 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.990476    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15198d8d-3540-4e9e-b4d3-de94a67aa733-plugins-config-volume\") pod \"15198d8d-3540-4e9e-b4d3-de94a67aa733\" (UID: \"15198d8d-3540-4e9e-b4d3-de94a67aa733\") "
Sep 09 18:49:02.990796 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.990762    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ad5922-1a81-4600-bc40-a409d1204c9c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40ad5922-1a81-4600-bc40-a409d1204c9c" (UID: "40ad5922-1a81-4600-bc40-a409d1204c9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:49:02.990939 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.990865    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15198d8d-3540-4e9e-b4d3-de94a67aa733-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15198d8d-3540-4e9e-b4d3-de94a67aa733" (UID: "15198d8d-3540-4e9e-b4d3-de94a67aa733"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:49:02.992841 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.992812    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ad5922-1a81-4600-bc40-a409d1204c9c-kube-api-access-5pkrz" (OuterVolumeSpecName: "kube-api-access-5pkrz") pod "40ad5922-1a81-4600-bc40-a409d1204c9c" (UID: "40ad5922-1a81-4600-bc40-a409d1204c9c"). InnerVolumeSpecName "kube-api-access-5pkrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:49:02.992931 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:02.992863    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15198d8d-3540-4e9e-b4d3-de94a67aa733-kube-api-access-2krxz" (OuterVolumeSpecName: "kube-api-access-2krxz") pod "15198d8d-3540-4e9e-b4d3-de94a67aa733" (UID: "15198d8d-3540-4e9e-b4d3-de94a67aa733"). InnerVolumeSpecName "kube-api-access-2krxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:49:03.092051 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:03.092008    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pkrz\" (UniqueName: \"kubernetes.io/projected/40ad5922-1a81-4600-bc40-a409d1204c9c-kube-api-access-5pkrz\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:49:03.092051 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:03.092045    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15198d8d-3540-4e9e-b4d3-de94a67aa733-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:49:03.092051 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:03.092055    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ad5922-1a81-4600-bc40-a409d1204c9c-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:49:03.092308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:03.092063    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2krxz\" (UniqueName: \"kubernetes.io/projected/15198d8d-3540-4e9e-b4d3-de94a67aa733-kube-api-access-2krxz\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:49:03.309892 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:03.309854    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"]
Sep 09 18:49:03.316399 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:03.316364    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d0c2fa5c-846779554b-2xnks"]
Sep 09 18:49:03.352902 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:03.352861    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"]
Sep 09 18:49:03.359098 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:03.359060    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d0c2fa5c-b8bbc6fd7-sc7ps"]
Sep 09 18:49:04.321075 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:04.321039    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15198d8d-3540-4e9e-b4d3-de94a67aa733" path="/var/lib/kubelet/pods/15198d8d-3540-4e9e-b4d3-de94a67aa733/volumes"
Sep 09 18:49:04.321531 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:04.321440    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40ad5922-1a81-4600-bc40-a409d1204c9c" path="/var/lib/kubelet/pods/40ad5922-1a81-4600-bc40-a409d1204c9c/volumes"
Sep 09 18:49:07.992724 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:07.992680    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"]
Sep 09 18:49:07.993125 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:07.993112    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15198d8d-3540-4e9e-b4d3-de94a67aa733" containerName="payload-pre-processing"
Sep 09 18:49:07.993125 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:07.993126    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="15198d8d-3540-4e9e-b4d3-de94a67aa733" containerName="payload-pre-processing"
Sep 09 18:49:07.993221 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:07.993144    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40ad5922-1a81-4600-bc40-a409d1204c9c" containerName="payload-processing"
Sep 09 18:49:07.993221 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:07.993150    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="40ad5922-1a81-4600-bc40-a409d1204c9c" containerName="payload-processing"
Sep 09 18:49:07.993221 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:07.993216    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="15198d8d-3540-4e9e-b4d3-de94a67aa733" containerName="payload-pre-processing"
Sep 09 18:49:07.993310 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:07.993227    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="40ad5922-1a81-4600-bc40-a409d1204c9c" containerName="payload-processing"
Sep 09 18:49:07.996948 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:07.996925    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"
Sep 09 18:49:07.999624 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:07.999598    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b6a21af8-dockercfg-69snr\""
Sep 09 18:49:07.999787 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:07.999631    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b6a21af8\""
Sep 09 18:49:08.009588 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:08.009554    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"]
Sep 09 18:49:08.044782 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:08.044728    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543a26ed-a9ac-4f4e-b65f-b01624e38015-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv\" (UID: \"543a26ed-a9ac-4f4e-b65f-b01624e38015\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"
Sep 09 18:49:08.044972 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:08.044903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vg24\" (UniqueName: \"kubernetes.io/projected/543a26ed-a9ac-4f4e-b65f-b01624e38015-kube-api-access-2vg24\") pod \"payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv\" (UID: \"543a26ed-a9ac-4f4e-b65f-b01624e38015\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"
Sep 09 18:49:08.146266 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:08.146207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vg24\" (UniqueName: \"kubernetes.io/projected/543a26ed-a9ac-4f4e-b65f-b01624e38015-kube-api-access-2vg24\") pod \"payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv\" (UID: \"543a26ed-a9ac-4f4e-b65f-b01624e38015\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"
Sep 09 18:49:08.146543 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:08.146300    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543a26ed-a9ac-4f4e-b65f-b01624e38015-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv\" (UID: \"543a26ed-a9ac-4f4e-b65f-b01624e38015\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"
Sep 09 18:49:08.147066 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:08.147039    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543a26ed-a9ac-4f4e-b65f-b01624e38015-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv\" (UID: \"543a26ed-a9ac-4f4e-b65f-b01624e38015\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"
Sep 09 18:49:08.154983 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:08.154951    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vg24\" (UniqueName: \"kubernetes.io/projected/543a26ed-a9ac-4f4e-b65f-b01624e38015-kube-api-access-2vg24\") pod \"payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv\" (UID: \"543a26ed-a9ac-4f4e-b65f-b01624e38015\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"
Sep 09 18:49:08.308982 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:08.308880    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"
Sep 09 18:49:08.583764 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:08.583733    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"]
Sep 09 18:49:08.586245 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:49:08.586202    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543a26ed_a9ac_4f4e_b65f_b01624e38015.slice/crio-da2d87c0df61a4ceee606c5ea2c68de19153bfabf13b31742ed948ab671d6857 WatchSource:0}: Error finding container da2d87c0df61a4ceee606c5ea2c68de19153bfabf13b31742ed948ab671d6857: Status 404 returned error can't find the container with id da2d87c0df61a4ceee606c5ea2c68de19153bfabf13b31742ed948ab671d6857
Sep 09 18:49:08.994449 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:08.994409    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv" event={"ID":"543a26ed-a9ac-4f4e-b65f-b01624e38015","Type":"ContainerStarted","Data":"27de79f71b06f1d746710bc66596454e5e868671b608c628e4732f57f4fa5baf"}
Sep 09 18:49:08.995112 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:08.994464    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv" event={"ID":"543a26ed-a9ac-4f4e-b65f-b01624e38015","Type":"ContainerStarted","Data":"da2d87c0df61a4ceee606c5ea2c68de19153bfabf13b31742ed948ab671d6857"}
Sep 09 18:49:08.995112 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:08.994561    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"
Sep 09 18:49:09.012708 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:09.012642    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv" podStartSLOduration=2.012621211 podStartE2EDuration="2.012621211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:49:09.01113249 +0000 UTC m=+1037.248502306" watchObservedRunningTime="2026-09-09 18:49:09.012621211 +0000 UTC m=+1037.249991025"
Sep 09 18:49:13.829587 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:13.829544    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"]
Sep 09 18:49:13.832326 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:13.832306    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:13.835072 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:13.835048    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-8b6cce27-kserve-self-signed-certs\""
Sep 09 18:49:13.842299 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:13.842267    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"]
Sep 09 18:49:13.904801 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:13.904771    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-tls-certs\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:13.904801 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:13.904804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-model-cache\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:13.905017 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:13.904831    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-tmp-dir\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:13.905017 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:13.904933    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-dshm\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:13.905017 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:13.904989    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-home\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:13.905118 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:13.905015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rxqd\" (UniqueName: \"kubernetes.io/projected/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-kube-api-access-8rxqd\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:14.006277 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:14.006230    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-tls-certs\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:14.006481 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:14.006290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-model-cache\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:14.006481 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:14.006332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-tmp-dir\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:14.006481 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:14.006400    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-dshm\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:14.006481 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:14.006462    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-home\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:14.006727 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:14.006516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rxqd\" (UniqueName: \"kubernetes.io/projected/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-kube-api-access-8rxqd\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:14.006848 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:14.006821    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-model-cache\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:14.006922 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:14.006865    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-tmp-dir\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:14.007124 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:14.007099    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-home\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:14.008767 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:14.008743    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-dshm\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:14.009407 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:14.009383    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-tls-certs\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:14.015595 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:14.015561    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rxqd\" (UniqueName: \"kubernetes.io/projected/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-kube-api-access-8rxqd\") pod \"e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") " pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:14.143461 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:14.143417    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:14.423410 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:14.423369    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"]
Sep 09 18:49:14.424543 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:49:14.424481    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6632e79_814d_4da9_9cb5_9fbd6f4f6217.slice/crio-c89c0760591e0670ef49a3c3b8bf26129025bd563a28f0de0d06cb1f7f644650 WatchSource:0}: Error finding container c89c0760591e0670ef49a3c3b8bf26129025bd563a28f0de0d06cb1f7f644650: Status 404 returned error can't find the container with id c89c0760591e0670ef49a3c3b8bf26129025bd563a28f0de0d06cb1f7f644650
Sep 09 18:49:15.031859 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:15.031816    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx" event={"ID":"f6632e79-814d-4da9-9cb5-9fbd6f4f6217","Type":"ContainerStarted","Data":"2f62381e2c14b0de80be434029c5771b832c98c660816057b55c34faac3d4526"}
Sep 09 18:49:15.032280 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:15.031870    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx" event={"ID":"f6632e79-814d-4da9-9cb5-9fbd6f4f6217","Type":"ContainerStarted","Data":"c89c0760591e0670ef49a3c3b8bf26129025bd563a28f0de0d06cb1f7f644650"}
Sep 09 18:49:15.061712 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:15.061648    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx" podStartSLOduration=2.061629743 podStartE2EDuration="2.061629743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:49:15.060298452 +0000 UTC m=+1043.297668266" watchObservedRunningTime="2026-09-09 18:49:15.061629743 +0000 UTC m=+1043.298999558"
Sep 09 18:49:20.003083 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:20.003047    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"
Sep 09 18:49:24.144020 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:24.143979    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:24.144547 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:24.144260    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:24.157028 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:24.156995    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:25.097699 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:25.097666    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:49:38.807978 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:38.807938    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"]
Sep 09 18:49:38.808779 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:38.808723    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv" podUID="543a26ed-a9ac-4f4e-b65f-b01624e38015" containerName="payload-pre-processing" containerID="cri-o://27de79f71b06f1d746710bc66596454e5e868671b608c628e4732f57f4fa5baf" gracePeriod=30
Sep 09 18:49:39.074828 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.074754    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"
Sep 09 18:49:39.158874 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.158841    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vg24\" (UniqueName: \"kubernetes.io/projected/543a26ed-a9ac-4f4e-b65f-b01624e38015-kube-api-access-2vg24\") pod \"543a26ed-a9ac-4f4e-b65f-b01624e38015\" (UID: \"543a26ed-a9ac-4f4e-b65f-b01624e38015\") "
Sep 09 18:49:39.159058 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.158884    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543a26ed-a9ac-4f4e-b65f-b01624e38015-plugins-config-volume\") pod \"543a26ed-a9ac-4f4e-b65f-b01624e38015\" (UID: \"543a26ed-a9ac-4f4e-b65f-b01624e38015\") "
Sep 09 18:49:39.159325 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.159299    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543a26ed-a9ac-4f4e-b65f-b01624e38015-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "543a26ed-a9ac-4f4e-b65f-b01624e38015" (UID: "543a26ed-a9ac-4f4e-b65f-b01624e38015"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:49:39.161093 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.161065    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543a26ed-a9ac-4f4e-b65f-b01624e38015-kube-api-access-2vg24" (OuterVolumeSpecName: "kube-api-access-2vg24") pod "543a26ed-a9ac-4f4e-b65f-b01624e38015" (UID: "543a26ed-a9ac-4f4e-b65f-b01624e38015"). InnerVolumeSpecName "kube-api-access-2vg24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:49:39.169348 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.169317    2575 generic.go:356] "Generic (PLEG): container finished" podID="543a26ed-a9ac-4f4e-b65f-b01624e38015" containerID="27de79f71b06f1d746710bc66596454e5e868671b608c628e4732f57f4fa5baf" exitCode=0
Sep 09 18:49:39.169459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.169365    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"
Sep 09 18:49:39.169459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.169379    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv" event={"ID":"543a26ed-a9ac-4f4e-b65f-b01624e38015","Type":"ContainerDied","Data":"27de79f71b06f1d746710bc66596454e5e868671b608c628e4732f57f4fa5baf"}
Sep 09 18:49:39.169459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.169434    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv" event={"ID":"543a26ed-a9ac-4f4e-b65f-b01624e38015","Type":"ContainerDied","Data":"da2d87c0df61a4ceee606c5ea2c68de19153bfabf13b31742ed948ab671d6857"}
Sep 09 18:49:39.169630 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.169464    2575 scope.go:117] "RemoveContainer" containerID="27de79f71b06f1d746710bc66596454e5e868671b608c628e4732f57f4fa5baf"
Sep 09 18:49:39.179330 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.179308    2575 scope.go:117] "RemoveContainer" containerID="27de79f71b06f1d746710bc66596454e5e868671b608c628e4732f57f4fa5baf"
Sep 09 18:49:39.179772 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:49:39.179750    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27de79f71b06f1d746710bc66596454e5e868671b608c628e4732f57f4fa5baf\": container with ID starting with 27de79f71b06f1d746710bc66596454e5e868671b608c628e4732f57f4fa5baf not found: ID does not exist" containerID="27de79f71b06f1d746710bc66596454e5e868671b608c628e4732f57f4fa5baf"
Sep 09 18:49:39.179827 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.179782    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27de79f71b06f1d746710bc66596454e5e868671b608c628e4732f57f4fa5baf"} err="failed to get container status \"27de79f71b06f1d746710bc66596454e5e868671b608c628e4732f57f4fa5baf\": rpc error: code = NotFound desc = could not find container \"27de79f71b06f1d746710bc66596454e5e868671b608c628e4732f57f4fa5baf\": container with ID starting with 27de79f71b06f1d746710bc66596454e5e868671b608c628e4732f57f4fa5baf not found: ID does not exist"
Sep 09 18:49:39.193798 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.193766    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"]
Sep 09 18:49:39.197793 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.197762    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b6a21af8-7f8f8f5b66-hqppv"]
Sep 09 18:49:39.259953 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.259918    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vg24\" (UniqueName: \"kubernetes.io/projected/543a26ed-a9ac-4f4e-b65f-b01624e38015-kube-api-access-2vg24\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:49:39.259953 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:39.259946    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543a26ed-a9ac-4f4e-b65f-b01624e38015-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:49:40.321956 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:40.321915    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="543a26ed-a9ac-4f4e-b65f-b01624e38015" path="/var/lib/kubelet/pods/543a26ed-a9ac-4f4e-b65f-b01624e38015/volumes"
Sep 09 18:49:44.519121 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.519038    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"]
Sep 09 18:49:44.519567 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.519464    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="543a26ed-a9ac-4f4e-b65f-b01624e38015" containerName="payload-pre-processing"
Sep 09 18:49:44.519567 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.519475    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="543a26ed-a9ac-4f4e-b65f-b01624e38015" containerName="payload-pre-processing"
Sep 09 18:49:44.519567 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.519555    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="543a26ed-a9ac-4f4e-b65f-b01624e38015" containerName="payload-pre-processing"
Sep 09 18:49:44.524133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.524113    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"
Sep 09 18:49:44.527560 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.527530    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-65f3bfa3-dockercfg-wbsvp\""
Sep 09 18:49:44.527723 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.527531    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-65f3bfa3\""
Sep 09 18:49:44.538182 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.538145    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"]
Sep 09 18:49:44.617892 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.617856    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zw4n\" (UniqueName: \"kubernetes.io/projected/d73c66d8-7be4-4537-af20-2276041f79eb-kube-api-access-9zw4n\") pod \"payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx\" (UID: \"d73c66d8-7be4-4537-af20-2276041f79eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"
Sep 09 18:49:44.618076 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.617902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73c66d8-7be4-4537-af20-2276041f79eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx\" (UID: \"d73c66d8-7be4-4537-af20-2276041f79eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"
Sep 09 18:49:44.624991 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.624959    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"]
Sep 09 18:49:44.628368 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.628346    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"
Sep 09 18:49:44.647025 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.646991    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"]
Sep 09 18:49:44.719182 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.719139    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73c66d8-7be4-4537-af20-2276041f79eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx\" (UID: \"d73c66d8-7be4-4537-af20-2276041f79eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"
Sep 09 18:49:44.719365 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.719228    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef08548c-f999-48bc-ae35-9176588ef82c-plugins-config-volume\") pod \"payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh\" (UID: \"ef08548c-f999-48bc-ae35-9176588ef82c\") " pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"
Sep 09 18:49:44.719365 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.719297    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zw4n\" (UniqueName: \"kubernetes.io/projected/d73c66d8-7be4-4537-af20-2276041f79eb-kube-api-access-9zw4n\") pod \"payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx\" (UID: \"d73c66d8-7be4-4537-af20-2276041f79eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"
Sep 09 18:49:44.719365 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.719322    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nqbz\" (UniqueName: \"kubernetes.io/projected/ef08548c-f999-48bc-ae35-9176588ef82c-kube-api-access-7nqbz\") pod \"payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh\" (UID: \"ef08548c-f999-48bc-ae35-9176588ef82c\") " pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"
Sep 09 18:49:44.719959 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.719938    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73c66d8-7be4-4537-af20-2276041f79eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx\" (UID: \"d73c66d8-7be4-4537-af20-2276041f79eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"
Sep 09 18:49:44.729476 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.729443    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zw4n\" (UniqueName: \"kubernetes.io/projected/d73c66d8-7be4-4537-af20-2276041f79eb-kube-api-access-9zw4n\") pod \"payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx\" (UID: \"d73c66d8-7be4-4537-af20-2276041f79eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"
Sep 09 18:49:44.820877 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.820778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nqbz\" (UniqueName: \"kubernetes.io/projected/ef08548c-f999-48bc-ae35-9176588ef82c-kube-api-access-7nqbz\") pod \"payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh\" (UID: \"ef08548c-f999-48bc-ae35-9176588ef82c\") " pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"
Sep 09 18:49:44.821072 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.820886    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef08548c-f999-48bc-ae35-9176588ef82c-plugins-config-volume\") pod \"payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh\" (UID: \"ef08548c-f999-48bc-ae35-9176588ef82c\") " pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"
Sep 09 18:49:44.821821 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.821793    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef08548c-f999-48bc-ae35-9176588ef82c-plugins-config-volume\") pod \"payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh\" (UID: \"ef08548c-f999-48bc-ae35-9176588ef82c\") " pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"
Sep 09 18:49:44.831359 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.831313    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nqbz\" (UniqueName: \"kubernetes.io/projected/ef08548c-f999-48bc-ae35-9176588ef82c-kube-api-access-7nqbz\") pod \"payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh\" (UID: \"ef08548c-f999-48bc-ae35-9176588ef82c\") " pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"
Sep 09 18:49:44.835142 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.835116    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"
Sep 09 18:49:44.939139 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:44.939102    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"
Sep 09 18:49:45.120338 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:45.120305    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"]
Sep 09 18:49:45.122161 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:49:45.122129    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73c66d8_7be4_4537_af20_2276041f79eb.slice/crio-fa78b4fe321129f28a2249f1f99c8559649ce8d4ef73791da0fac9a7df4a28f4 WatchSource:0}: Error finding container fa78b4fe321129f28a2249f1f99c8559649ce8d4ef73791da0fac9a7df4a28f4: Status 404 returned error can't find the container with id fa78b4fe321129f28a2249f1f99c8559649ce8d4ef73791da0fac9a7df4a28f4
Sep 09 18:49:45.216715 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:45.216632    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx" event={"ID":"d73c66d8-7be4-4537-af20-2276041f79eb","Type":"ContainerStarted","Data":"b5ac61a6f4250c98e405da8e953f883d9c2ee99ef9d4ea577e9a889eb9f4ecf2"}
Sep 09 18:49:45.216715 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:45.216834    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"
Sep 09 18:49:45.216715 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:45.216867    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx" event={"ID":"d73c66d8-7be4-4537-af20-2276041f79eb","Type":"ContainerStarted","Data":"fa78b4fe321129f28a2249f1f99c8559649ce8d4ef73791da0fac9a7df4a28f4"}
Sep 09 18:49:45.220795 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:45.220747    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"]
Sep 09 18:49:45.222672 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:49:45.222644    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef08548c_f999_48bc_ae35_9176588ef82c.slice/crio-3ea3bcc5dfdde9416eb48d82da819d6163aad74f0d0bc9d4463ae81fe7e0d128 WatchSource:0}: Error finding container 3ea3bcc5dfdde9416eb48d82da819d6163aad74f0d0bc9d4463ae81fe7e0d128: Status 404 returned error can't find the container with id 3ea3bcc5dfdde9416eb48d82da819d6163aad74f0d0bc9d4463ae81fe7e0d128
Sep 09 18:49:45.243910 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:45.243864    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx" podStartSLOduration=1.243844658 podStartE2EDuration="1.243844658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:49:45.241677673 +0000 UTC m=+1073.479047518" watchObservedRunningTime="2026-09-09 18:49:45.243844658 +0000 UTC m=+1073.481214487"
Sep 09 18:49:46.223428 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:46.223385    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh" event={"ID":"ef08548c-f999-48bc-ae35-9176588ef82c","Type":"ContainerStarted","Data":"92a84f98380183595cda441db5d4c08cb1fbba80f3ede43301015ed3b772f43f"}
Sep 09 18:49:46.223975 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:46.223685    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"
Sep 09 18:49:46.223975 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:46.223725    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh" event={"ID":"ef08548c-f999-48bc-ae35-9176588ef82c","Type":"ContainerStarted","Data":"3ea3bcc5dfdde9416eb48d82da819d6163aad74f0d0bc9d4463ae81fe7e0d128"}
Sep 09 18:49:46.242948 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:46.242889    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh" podStartSLOduration=2.242870014 podStartE2EDuration="2.242870014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:49:46.241686035 +0000 UTC m=+1074.479055852" watchObservedRunningTime="2026-09-09 18:49:46.242870014 +0000 UTC m=+1074.480239829"
Sep 09 18:49:46.934998 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:46.934956    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"]
Sep 09 18:49:56.226126 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:56.226092    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"
Sep 09 18:49:57.230771 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:49:57.230734    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"
Sep 09 18:50:01.955633 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:01.955565    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx" podUID="f6632e79-814d-4da9-9cb5-9fbd6f4f6217" containerName="main" containerID="cri-o://2f62381e2c14b0de80be434029c5771b832c98c660816057b55c34faac3d4526" gracePeriod=45
Sep 09 18:50:02.212115 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.212045    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:50:02.313169 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.313137    2575 generic.go:356] "Generic (PLEG): container finished" podID="f6632e79-814d-4da9-9cb5-9fbd6f4f6217" containerID="2f62381e2c14b0de80be434029c5771b832c98c660816057b55c34faac3d4526" exitCode=0
Sep 09 18:50:02.313354 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.313182    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"
Sep 09 18:50:02.313354 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.313232    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx" event={"ID":"f6632e79-814d-4da9-9cb5-9fbd6f4f6217","Type":"ContainerDied","Data":"2f62381e2c14b0de80be434029c5771b832c98c660816057b55c34faac3d4526"}
Sep 09 18:50:02.313354 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.313277    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx" event={"ID":"f6632e79-814d-4da9-9cb5-9fbd6f4f6217","Type":"ContainerDied","Data":"c89c0760591e0670ef49a3c3b8bf26129025bd563a28f0de0d06cb1f7f644650"}
Sep 09 18:50:02.313354 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.313295    2575 scope.go:117] "RemoveContainer" containerID="2f62381e2c14b0de80be434029c5771b832c98c660816057b55c34faac3d4526"
Sep 09 18:50:02.330200 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.330174    2575 scope.go:117] "RemoveContainer" containerID="2f62381e2c14b0de80be434029c5771b832c98c660816057b55c34faac3d4526"
Sep 09 18:50:02.330562 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:50:02.330540    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f62381e2c14b0de80be434029c5771b832c98c660816057b55c34faac3d4526\": container with ID starting with 2f62381e2c14b0de80be434029c5771b832c98c660816057b55c34faac3d4526 not found: ID does not exist" containerID="2f62381e2c14b0de80be434029c5771b832c98c660816057b55c34faac3d4526"
Sep 09 18:50:02.330606 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.330573    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f62381e2c14b0de80be434029c5771b832c98c660816057b55c34faac3d4526"} err="failed to get container status \"2f62381e2c14b0de80be434029c5771b832c98c660816057b55c34faac3d4526\": rpc error: code = NotFound desc = could not find container \"2f62381e2c14b0de80be434029c5771b832c98c660816057b55c34faac3d4526\": container with ID starting with 2f62381e2c14b0de80be434029c5771b832c98c660816057b55c34faac3d4526 not found: ID does not exist"
Sep 09 18:50:02.403080 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.403046    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-home\") pod \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") "
Sep 09 18:50:02.403251 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.403093    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-model-cache\") pod \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") "
Sep 09 18:50:02.403251 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.403127    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-dshm\") pod \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") "
Sep 09 18:50:02.403251 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.403201    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-tmp-dir\") pod \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") "
Sep 09 18:50:02.403251 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.403230    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rxqd\" (UniqueName: \"kubernetes.io/projected/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-kube-api-access-8rxqd\") pod \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") "
Sep 09 18:50:02.403470 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.403281    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-tls-certs\") pod \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\" (UID: \"f6632e79-814d-4da9-9cb5-9fbd6f4f6217\") "
Sep 09 18:50:02.403470 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.403318    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-home" (OuterVolumeSpecName: "home") pod "f6632e79-814d-4da9-9cb5-9fbd6f4f6217" (UID: "f6632e79-814d-4da9-9cb5-9fbd6f4f6217"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:50:02.403470 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.403402    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6632e79-814d-4da9-9cb5-9fbd6f4f6217" (UID: "f6632e79-814d-4da9-9cb5-9fbd6f4f6217"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:50:02.403671 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.403527    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6632e79-814d-4da9-9cb5-9fbd6f4f6217" (UID: "f6632e79-814d-4da9-9cb5-9fbd6f4f6217"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:50:02.403741 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.403718    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:02.403741 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.403737    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:02.403872 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.403746    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:02.405612 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.405577    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-dshm" (OuterVolumeSpecName: "dshm") pod "f6632e79-814d-4da9-9cb5-9fbd6f4f6217" (UID: "f6632e79-814d-4da9-9cb5-9fbd6f4f6217"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:50:02.405726 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.405647    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6632e79-814d-4da9-9cb5-9fbd6f4f6217" (UID: "f6632e79-814d-4da9-9cb5-9fbd6f4f6217"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:50:02.405726 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.405676    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-kube-api-access-8rxqd" (OuterVolumeSpecName: "kube-api-access-8rxqd") pod "f6632e79-814d-4da9-9cb5-9fbd6f4f6217" (UID: "f6632e79-814d-4da9-9cb5-9fbd6f4f6217"). InnerVolumeSpecName "kube-api-access-8rxqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:50:02.505363 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.505265    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rxqd\" (UniqueName: \"kubernetes.io/projected/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-kube-api-access-8rxqd\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:02.505363 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.505294    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:02.505363 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.505305    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6632e79-814d-4da9-9cb5-9fbd6f4f6217-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:02.638887 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.638850    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"]
Sep 09 18:50:02.640809 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:02.640784    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-8b6cce27-kserve-789497984f-tmndx"]
Sep 09 18:50:04.322148 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:04.322108    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6632e79-814d-4da9-9cb5-9fbd6f4f6217" path="/var/lib/kubelet/pods/f6632e79-814d-4da9-9cb5-9fbd6f4f6217/volumes"
Sep 09 18:50:04.696858 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:04.696815    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"]
Sep 09 18:50:04.697125 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:04.697073    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq" podUID="6799af4e-e6fc-43bc-9423-082e9429ae9f" containerName="payload-processing" containerID="cri-o://b5b173fa819578769e1ce55091b3d2900e9e8cf6f474c7331f63f9521006cd63" gracePeriod=30
Sep 09 18:50:04.968622 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:04.968589    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"
Sep 09 18:50:05.030628 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.030594    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6799af4e-e6fc-43bc-9423-082e9429ae9f-plugins-config-volume\") pod \"6799af4e-e6fc-43bc-9423-082e9429ae9f\" (UID: \"6799af4e-e6fc-43bc-9423-082e9429ae9f\") "
Sep 09 18:50:05.030813 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.030655    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk5v2\" (UniqueName: \"kubernetes.io/projected/6799af4e-e6fc-43bc-9423-082e9429ae9f-kube-api-access-xk5v2\") pod \"6799af4e-e6fc-43bc-9423-082e9429ae9f\" (UID: \"6799af4e-e6fc-43bc-9423-082e9429ae9f\") "
Sep 09 18:50:05.031081 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.031046    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6799af4e-e6fc-43bc-9423-082e9429ae9f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6799af4e-e6fc-43bc-9423-082e9429ae9f" (UID: "6799af4e-e6fc-43bc-9423-082e9429ae9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:50:05.032935 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.032898    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6799af4e-e6fc-43bc-9423-082e9429ae9f-kube-api-access-xk5v2" (OuterVolumeSpecName: "kube-api-access-xk5v2") pod "6799af4e-e6fc-43bc-9423-082e9429ae9f" (UID: "6799af4e-e6fc-43bc-9423-082e9429ae9f"). InnerVolumeSpecName "kube-api-access-xk5v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:50:05.131662 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.131620    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk5v2\" (UniqueName: \"kubernetes.io/projected/6799af4e-e6fc-43bc-9423-082e9429ae9f-kube-api-access-xk5v2\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:05.131662 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.131654    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6799af4e-e6fc-43bc-9423-082e9429ae9f-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:05.331454 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.331360    2575 generic.go:356] "Generic (PLEG): container finished" podID="6799af4e-e6fc-43bc-9423-082e9429ae9f" containerID="b5b173fa819578769e1ce55091b3d2900e9e8cf6f474c7331f63f9521006cd63" exitCode=0
Sep 09 18:50:05.331454 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.331424    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"
Sep 09 18:50:05.331454 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.331435    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq" event={"ID":"6799af4e-e6fc-43bc-9423-082e9429ae9f","Type":"ContainerDied","Data":"b5b173fa819578769e1ce55091b3d2900e9e8cf6f474c7331f63f9521006cd63"}
Sep 09 18:50:05.332028 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.331483    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq" event={"ID":"6799af4e-e6fc-43bc-9423-082e9429ae9f","Type":"ContainerDied","Data":"4e09ac0928225bab8b4d5d3ccede0a4d41d539b7893dc5c81c68091d58e7196d"}
Sep 09 18:50:05.332028 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.331523    2575 scope.go:117] "RemoveContainer" containerID="b5b173fa819578769e1ce55091b3d2900e9e8cf6f474c7331f63f9521006cd63"
Sep 09 18:50:05.341729 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.341698    2575 scope.go:117] "RemoveContainer" containerID="b5b173fa819578769e1ce55091b3d2900e9e8cf6f474c7331f63f9521006cd63"
Sep 09 18:50:05.342114 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:50:05.342089    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b173fa819578769e1ce55091b3d2900e9e8cf6f474c7331f63f9521006cd63\": container with ID starting with b5b173fa819578769e1ce55091b3d2900e9e8cf6f474c7331f63f9521006cd63 not found: ID does not exist" containerID="b5b173fa819578769e1ce55091b3d2900e9e8cf6f474c7331f63f9521006cd63"
Sep 09 18:50:05.342171 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.342129    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b173fa819578769e1ce55091b3d2900e9e8cf6f474c7331f63f9521006cd63"} err="failed to get container status \"b5b173fa819578769e1ce55091b3d2900e9e8cf6f474c7331f63f9521006cd63\": rpc error: code = NotFound desc = could not find container \"b5b173fa819578769e1ce55091b3d2900e9e8cf6f474c7331f63f9521006cd63\": container with ID starting with b5b173fa819578769e1ce55091b3d2900e9e8cf6f474c7331f63f9521006cd63 not found: ID does not exist"
Sep 09 18:50:05.356849 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.356807    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"]
Sep 09 18:50:05.363148 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:05.363101    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-38af9d-7f558c645f-fm8bq"]
Sep 09 18:50:06.322113 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:06.322073    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6799af4e-e6fc-43bc-9423-082e9429ae9f" path="/var/lib/kubelet/pods/6799af4e-e6fc-43bc-9423-082e9429ae9f/volumes"
Sep 09 18:50:12.835858 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:12.835812    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"]
Sep 09 18:50:14.323144 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.323109    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"]
Sep 09 18:50:14.323676 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.323365    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh" podUID="ef08548c-f999-48bc-ae35-9176588ef82c" containerName="payload-processing" containerID="cri-o://92a84f98380183595cda441db5d4c08cb1fbba80f3ede43301015ed3b772f43f" gracePeriod=30
Sep 09 18:50:14.326075 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.325442    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"]
Sep 09 18:50:14.326336 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.326295    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx" podUID="d73c66d8-7be4-4537-af20-2276041f79eb" containerName="payload-pre-processing" containerID="cri-o://b5ac61a6f4250c98e405da8e953f883d9c2ee99ef9d4ea577e9a889eb9f4ecf2" gracePeriod=30
Sep 09 18:50:14.613615 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.613589    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"
Sep 09 18:50:14.617042 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.617015    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"
Sep 09 18:50:14.727947 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.727902    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nqbz\" (UniqueName: \"kubernetes.io/projected/ef08548c-f999-48bc-ae35-9176588ef82c-kube-api-access-7nqbz\") pod \"ef08548c-f999-48bc-ae35-9176588ef82c\" (UID: \"ef08548c-f999-48bc-ae35-9176588ef82c\") "
Sep 09 18:50:14.728133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.728008    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zw4n\" (UniqueName: \"kubernetes.io/projected/d73c66d8-7be4-4537-af20-2276041f79eb-kube-api-access-9zw4n\") pod \"d73c66d8-7be4-4537-af20-2276041f79eb\" (UID: \"d73c66d8-7be4-4537-af20-2276041f79eb\") "
Sep 09 18:50:14.728133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.728107    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73c66d8-7be4-4537-af20-2276041f79eb-plugins-config-volume\") pod \"d73c66d8-7be4-4537-af20-2276041f79eb\" (UID: \"d73c66d8-7be4-4537-af20-2276041f79eb\") "
Sep 09 18:50:14.728281 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.728210    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef08548c-f999-48bc-ae35-9176588ef82c-plugins-config-volume\") pod \"ef08548c-f999-48bc-ae35-9176588ef82c\" (UID: \"ef08548c-f999-48bc-ae35-9176588ef82c\") "
Sep 09 18:50:14.728603 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.728573    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73c66d8-7be4-4537-af20-2276041f79eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d73c66d8-7be4-4537-af20-2276041f79eb" (UID: "d73c66d8-7be4-4537-af20-2276041f79eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:50:14.728714 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.728602    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef08548c-f999-48bc-ae35-9176588ef82c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ef08548c-f999-48bc-ae35-9176588ef82c" (UID: "ef08548c-f999-48bc-ae35-9176588ef82c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:50:14.730119 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.730095    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef08548c-f999-48bc-ae35-9176588ef82c-kube-api-access-7nqbz" (OuterVolumeSpecName: "kube-api-access-7nqbz") pod "ef08548c-f999-48bc-ae35-9176588ef82c" (UID: "ef08548c-f999-48bc-ae35-9176588ef82c"). InnerVolumeSpecName "kube-api-access-7nqbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:50:14.730204 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.730144    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73c66d8-7be4-4537-af20-2276041f79eb-kube-api-access-9zw4n" (OuterVolumeSpecName: "kube-api-access-9zw4n") pod "d73c66d8-7be4-4537-af20-2276041f79eb" (UID: "d73c66d8-7be4-4537-af20-2276041f79eb"). InnerVolumeSpecName "kube-api-access-9zw4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:50:14.829927 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.829882    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nqbz\" (UniqueName: \"kubernetes.io/projected/ef08548c-f999-48bc-ae35-9176588ef82c-kube-api-access-7nqbz\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:14.829927 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.829922    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zw4n\" (UniqueName: \"kubernetes.io/projected/d73c66d8-7be4-4537-af20-2276041f79eb-kube-api-access-9zw4n\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:14.829927 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.829934    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d73c66d8-7be4-4537-af20-2276041f79eb-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:14.830174 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:14.829946    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef08548c-f999-48bc-ae35-9176588ef82c-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:15.394199 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.394166    2575 generic.go:356] "Generic (PLEG): container finished" podID="d73c66d8-7be4-4537-af20-2276041f79eb" containerID="b5ac61a6f4250c98e405da8e953f883d9c2ee99ef9d4ea577e9a889eb9f4ecf2" exitCode=0
Sep 09 18:50:15.394693 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.394212    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"
Sep 09 18:50:15.394693 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.394247    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx" event={"ID":"d73c66d8-7be4-4537-af20-2276041f79eb","Type":"ContainerDied","Data":"b5ac61a6f4250c98e405da8e953f883d9c2ee99ef9d4ea577e9a889eb9f4ecf2"}
Sep 09 18:50:15.394693 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.394297    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx" event={"ID":"d73c66d8-7be4-4537-af20-2276041f79eb","Type":"ContainerDied","Data":"fa78b4fe321129f28a2249f1f99c8559649ce8d4ef73791da0fac9a7df4a28f4"}
Sep 09 18:50:15.394693 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.394320    2575 scope.go:117] "RemoveContainer" containerID="b5ac61a6f4250c98e405da8e953f883d9c2ee99ef9d4ea577e9a889eb9f4ecf2"
Sep 09 18:50:15.395996 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.395971    2575 generic.go:356] "Generic (PLEG): container finished" podID="ef08548c-f999-48bc-ae35-9176588ef82c" containerID="92a84f98380183595cda441db5d4c08cb1fbba80f3ede43301015ed3b772f43f" exitCode=0
Sep 09 18:50:15.396081 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.396017    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"
Sep 09 18:50:15.396081 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.396034    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh" event={"ID":"ef08548c-f999-48bc-ae35-9176588ef82c","Type":"ContainerDied","Data":"92a84f98380183595cda441db5d4c08cb1fbba80f3ede43301015ed3b772f43f"}
Sep 09 18:50:15.396149 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.396088    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh" event={"ID":"ef08548c-f999-48bc-ae35-9176588ef82c","Type":"ContainerDied","Data":"3ea3bcc5dfdde9416eb48d82da819d6163aad74f0d0bc9d4463ae81fe7e0d128"}
Sep 09 18:50:15.404637 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.404471    2575 scope.go:117] "RemoveContainer" containerID="b5ac61a6f4250c98e405da8e953f883d9c2ee99ef9d4ea577e9a889eb9f4ecf2"
Sep 09 18:50:15.404895 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:50:15.404871    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ac61a6f4250c98e405da8e953f883d9c2ee99ef9d4ea577e9a889eb9f4ecf2\": container with ID starting with b5ac61a6f4250c98e405da8e953f883d9c2ee99ef9d4ea577e9a889eb9f4ecf2 not found: ID does not exist" containerID="b5ac61a6f4250c98e405da8e953f883d9c2ee99ef9d4ea577e9a889eb9f4ecf2"
Sep 09 18:50:15.404961 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.404903    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ac61a6f4250c98e405da8e953f883d9c2ee99ef9d4ea577e9a889eb9f4ecf2"} err="failed to get container status \"b5ac61a6f4250c98e405da8e953f883d9c2ee99ef9d4ea577e9a889eb9f4ecf2\": rpc error: code = NotFound desc = could not find container \"b5ac61a6f4250c98e405da8e953f883d9c2ee99ef9d4ea577e9a889eb9f4ecf2\": container with ID starting with b5ac61a6f4250c98e405da8e953f883d9c2ee99ef9d4ea577e9a889eb9f4ecf2 not found: ID does not exist"
Sep 09 18:50:15.404961 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.404921    2575 scope.go:117] "RemoveContainer" containerID="92a84f98380183595cda441db5d4c08cb1fbba80f3ede43301015ed3b772f43f"
Sep 09 18:50:15.414063 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.414042    2575 scope.go:117] "RemoveContainer" containerID="92a84f98380183595cda441db5d4c08cb1fbba80f3ede43301015ed3b772f43f"
Sep 09 18:50:15.414373 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:50:15.414352    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a84f98380183595cda441db5d4c08cb1fbba80f3ede43301015ed3b772f43f\": container with ID starting with 92a84f98380183595cda441db5d4c08cb1fbba80f3ede43301015ed3b772f43f not found: ID does not exist" containerID="92a84f98380183595cda441db5d4c08cb1fbba80f3ede43301015ed3b772f43f"
Sep 09 18:50:15.414421 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.414384    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a84f98380183595cda441db5d4c08cb1fbba80f3ede43301015ed3b772f43f"} err="failed to get container status \"92a84f98380183595cda441db5d4c08cb1fbba80f3ede43301015ed3b772f43f\": rpc error: code = NotFound desc = could not find container \"92a84f98380183595cda441db5d4c08cb1fbba80f3ede43301015ed3b772f43f\": container with ID starting with 92a84f98380183595cda441db5d4c08cb1fbba80f3ede43301015ed3b772f43f not found: ID does not exist"
Sep 09 18:50:15.421824 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.421794    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"]
Sep 09 18:50:15.427106 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.427079    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-65f3bfa3-75c586f78-mt4sx"]
Sep 09 18:50:15.440471 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.440434    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"]
Sep 09 18:50:15.442861 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:15.442835    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-65f3bfa3-7cb9bb5b6c-slpvh"]
Sep 09 18:50:16.322977 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:16.322946    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d73c66d8-7be4-4537-af20-2276041f79eb" path="/var/lib/kubelet/pods/d73c66d8-7be4-4537-af20-2276041f79eb/volumes"
Sep 09 18:50:16.323340 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:16.323327    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef08548c-f999-48bc-ae35-9176588ef82c" path="/var/lib/kubelet/pods/ef08548c-f999-48bc-ae35-9176588ef82c/volumes"
Sep 09 18:50:18.476546 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.476487    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"]
Sep 09 18:50:18.477124 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.477102    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6799af4e-e6fc-43bc-9423-082e9429ae9f" containerName="payload-processing"
Sep 09 18:50:18.477212 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.477127    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6799af4e-e6fc-43bc-9423-082e9429ae9f" containerName="payload-processing"
Sep 09 18:50:18.477212 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.477158    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6632e79-814d-4da9-9cb5-9fbd6f4f6217" containerName="main"
Sep 09 18:50:18.477212 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.477166    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6632e79-814d-4da9-9cb5-9fbd6f4f6217" containerName="main"
Sep 09 18:50:18.477212 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.477186    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73c66d8-7be4-4537-af20-2276041f79eb" containerName="payload-pre-processing"
Sep 09 18:50:18.477212 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.477195    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73c66d8-7be4-4537-af20-2276041f79eb" containerName="payload-pre-processing"
Sep 09 18:50:18.477212 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.477209    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef08548c-f999-48bc-ae35-9176588ef82c" containerName="payload-processing"
Sep 09 18:50:18.477518 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.477218    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef08548c-f999-48bc-ae35-9176588ef82c" containerName="payload-processing"
Sep 09 18:50:18.477518 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.477337    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6799af4e-e6fc-43bc-9423-082e9429ae9f" containerName="payload-processing"
Sep 09 18:50:18.477518 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.477354    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6632e79-814d-4da9-9cb5-9fbd6f4f6217" containerName="main"
Sep 09 18:50:18.477518 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.477363    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d73c66d8-7be4-4537-af20-2276041f79eb" containerName="payload-pre-processing"
Sep 09 18:50:18.477518 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.477372    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef08548c-f999-48bc-ae35-9176588ef82c" containerName="payload-processing"
Sep 09 18:50:18.482124 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.482097    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"
Sep 09 18:50:18.484997 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.484963    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-f8eac9-dockercfg-zw8vc\""
Sep 09 18:50:18.485245 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.485230    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-f8eac9\""
Sep 09 18:50:18.498217 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.498183    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"]
Sep 09 18:50:18.569623 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.569583    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3eb9b8-6844-43b0-be35-f9a64e4a8218-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd\" (UID: \"af3eb9b8-6844-43b0-be35-f9a64e4a8218\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"
Sep 09 18:50:18.569792 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.569725    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kdfj\" (UniqueName: \"kubernetes.io/projected/af3eb9b8-6844-43b0-be35-f9a64e4a8218-kube-api-access-7kdfj\") pod \"payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd\" (UID: \"af3eb9b8-6844-43b0-be35-f9a64e4a8218\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"
Sep 09 18:50:18.596096 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.596060    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"]
Sep 09 18:50:18.599781 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.599763    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"
Sep 09 18:50:18.613336 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.613308    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"]
Sep 09 18:50:18.670476 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.670433    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3eb9b8-6844-43b0-be35-f9a64e4a8218-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd\" (UID: \"af3eb9b8-6844-43b0-be35-f9a64e4a8218\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"
Sep 09 18:50:18.670729 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.670602    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67634d09-6a9c-4182-83a1-f49b799201bc-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b\" (UID: \"67634d09-6a9c-4182-83a1-f49b799201bc\") " pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"
Sep 09 18:50:18.670729 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.670645    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kdfj\" (UniqueName: \"kubernetes.io/projected/af3eb9b8-6844-43b0-be35-f9a64e4a8218-kube-api-access-7kdfj\") pod \"payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd\" (UID: \"af3eb9b8-6844-43b0-be35-f9a64e4a8218\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"
Sep 09 18:50:18.670729 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.670703    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnh6b\" (UniqueName: \"kubernetes.io/projected/67634d09-6a9c-4182-83a1-f49b799201bc-kube-api-access-jnh6b\") pod \"payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b\" (UID: \"67634d09-6a9c-4182-83a1-f49b799201bc\") " pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"
Sep 09 18:50:18.671344 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.671321    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3eb9b8-6844-43b0-be35-f9a64e4a8218-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd\" (UID: \"af3eb9b8-6844-43b0-be35-f9a64e4a8218\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"
Sep 09 18:50:18.684844 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.684806    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kdfj\" (UniqueName: \"kubernetes.io/projected/af3eb9b8-6844-43b0-be35-f9a64e4a8218-kube-api-access-7kdfj\") pod \"payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd\" (UID: \"af3eb9b8-6844-43b0-be35-f9a64e4a8218\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"
Sep 09 18:50:18.771432 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.771322    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67634d09-6a9c-4182-83a1-f49b799201bc-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b\" (UID: \"67634d09-6a9c-4182-83a1-f49b799201bc\") " pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"
Sep 09 18:50:18.771432 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.771383    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnh6b\" (UniqueName: \"kubernetes.io/projected/67634d09-6a9c-4182-83a1-f49b799201bc-kube-api-access-jnh6b\") pod \"payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b\" (UID: \"67634d09-6a9c-4182-83a1-f49b799201bc\") " pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"
Sep 09 18:50:18.772110 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.772084    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67634d09-6a9c-4182-83a1-f49b799201bc-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b\" (UID: \"67634d09-6a9c-4182-83a1-f49b799201bc\") " pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"
Sep 09 18:50:18.781095 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.781065    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnh6b\" (UniqueName: \"kubernetes.io/projected/67634d09-6a9c-4182-83a1-f49b799201bc-kube-api-access-jnh6b\") pod \"payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b\" (UID: \"67634d09-6a9c-4182-83a1-f49b799201bc\") " pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"
Sep 09 18:50:18.792528 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.792482    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"
Sep 09 18:50:18.911229 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:18.911186    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"
Sep 09 18:50:19.070823 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:19.070791    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"]
Sep 09 18:50:19.072745 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:50:19.072721    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3eb9b8_6844_43b0_be35_f9a64e4a8218.slice/crio-4b56f6120298103df083cd31e9e980f885672afcf5b17191be27eb031a6a31d4 WatchSource:0}: Error finding container 4b56f6120298103df083cd31e9e980f885672afcf5b17191be27eb031a6a31d4: Status 404 returned error can't find the container with id 4b56f6120298103df083cd31e9e980f885672afcf5b17191be27eb031a6a31d4
Sep 09 18:50:19.195734 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:19.195696    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"]
Sep 09 18:50:19.199320 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:50:19.199282    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67634d09_6a9c_4182_83a1_f49b799201bc.slice/crio-ec8ccf5ed042634b2d27895f4a3b8cebe94eb94b7fd1a305b5bba474d6f52370 WatchSource:0}: Error finding container ec8ccf5ed042634b2d27895f4a3b8cebe94eb94b7fd1a305b5bba474d6f52370: Status 404 returned error can't find the container with id ec8ccf5ed042634b2d27895f4a3b8cebe94eb94b7fd1a305b5bba474d6f52370
Sep 09 18:50:19.423406 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:19.423366    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd" event={"ID":"af3eb9b8-6844-43b0-be35-f9a64e4a8218","Type":"ContainerStarted","Data":"792c65200d8bbf0c0927b2305b5a0d6a35e58942a10e417b21b40cb40f02fe5f"}
Sep 09 18:50:19.423660 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:19.423431    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd" event={"ID":"af3eb9b8-6844-43b0-be35-f9a64e4a8218","Type":"ContainerStarted","Data":"4b56f6120298103df083cd31e9e980f885672afcf5b17191be27eb031a6a31d4"}
Sep 09 18:50:19.423660 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:19.423534    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"
Sep 09 18:50:19.425123 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:19.425094    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b" event={"ID":"67634d09-6a9c-4182-83a1-f49b799201bc","Type":"ContainerStarted","Data":"cb6756ffe7c45438d3b7fdcaba8295e5acbd50f5e4ab81c7a6fba95954b5dbe9"}
Sep 09 18:50:19.425232 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:19.425147    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b" event={"ID":"67634d09-6a9c-4182-83a1-f49b799201bc","Type":"ContainerStarted","Data":"ec8ccf5ed042634b2d27895f4a3b8cebe94eb94b7fd1a305b5bba474d6f52370"}
Sep 09 18:50:19.425232 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:19.425181    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"
Sep 09 18:50:19.447765 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:19.447715    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd" podStartSLOduration=1.447697363 podStartE2EDuration="1.447697363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:50:19.444867467 +0000 UTC m=+1107.682237283" watchObservedRunningTime="2026-09-09 18:50:19.447697363 +0000 UTC m=+1107.685067178"
Sep 09 18:50:19.470567 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:19.470511    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b" podStartSLOduration=1.470478535 podStartE2EDuration="1.470478535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:50:19.468423538 +0000 UTC m=+1107.705793357" watchObservedRunningTime="2026-09-09 18:50:19.470478535 +0000 UTC m=+1107.707848350"
Sep 09 18:50:22.059153 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.059109    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"]
Sep 09 18:50:22.063258 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.063228    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"
Sep 09 18:50:22.068050 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.068006    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-edd73ecc-dockercfg-wjvkz\""
Sep 09 18:50:22.069154 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.069129    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-edd73ecc\""
Sep 09 18:50:22.090579 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.090541    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"]
Sep 09 18:50:22.177296 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.177258    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"]
Sep 09 18:50:22.181052 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.181024    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"
Sep 09 18:50:22.207577 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.207539    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f663d651-b894-4dcd-9391-726087715800-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8\" (UID: \"f663d651-b894-4dcd-9391-726087715800\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"
Sep 09 18:50:22.207735 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.207655    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rsrl\" (UniqueName: \"kubernetes.io/projected/f663d651-b894-4dcd-9391-726087715800-kube-api-access-9rsrl\") pod \"payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8\" (UID: \"f663d651-b894-4dcd-9391-726087715800\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"
Sep 09 18:50:22.221534 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.221481    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"]
Sep 09 18:50:22.308700 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.308660    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f663d651-b894-4dcd-9391-726087715800-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8\" (UID: \"f663d651-b894-4dcd-9391-726087715800\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"
Sep 09 18:50:22.308907 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.308736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7vjd\" (UniqueName: \"kubernetes.io/projected/80f37b1e-b948-45dc-b809-a7e6819afe8b-kube-api-access-f7vjd\") pod \"payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth\" (UID: \"80f37b1e-b948-45dc-b809-a7e6819afe8b\") " pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"
Sep 09 18:50:22.308907 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.308763    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rsrl\" (UniqueName: \"kubernetes.io/projected/f663d651-b894-4dcd-9391-726087715800-kube-api-access-9rsrl\") pod \"payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8\" (UID: \"f663d651-b894-4dcd-9391-726087715800\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"
Sep 09 18:50:22.309003 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.308931    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f37b1e-b948-45dc-b809-a7e6819afe8b-plugins-config-volume\") pod \"payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth\" (UID: \"80f37b1e-b948-45dc-b809-a7e6819afe8b\") " pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"
Sep 09 18:50:22.309480 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.309428    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f663d651-b894-4dcd-9391-726087715800-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8\" (UID: \"f663d651-b894-4dcd-9391-726087715800\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"
Sep 09 18:50:22.329574 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.329537    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rsrl\" (UniqueName: \"kubernetes.io/projected/f663d651-b894-4dcd-9391-726087715800-kube-api-access-9rsrl\") pod \"payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8\" (UID: \"f663d651-b894-4dcd-9391-726087715800\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"
Sep 09 18:50:22.376575 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.376529    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"
Sep 09 18:50:22.410074 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.410029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f37b1e-b948-45dc-b809-a7e6819afe8b-plugins-config-volume\") pod \"payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth\" (UID: \"80f37b1e-b948-45dc-b809-a7e6819afe8b\") " pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"
Sep 09 18:50:22.410260 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.410140    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vjd\" (UniqueName: \"kubernetes.io/projected/80f37b1e-b948-45dc-b809-a7e6819afe8b-kube-api-access-f7vjd\") pod \"payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth\" (UID: \"80f37b1e-b948-45dc-b809-a7e6819afe8b\") " pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"
Sep 09 18:50:22.410821 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.410799    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f37b1e-b948-45dc-b809-a7e6819afe8b-plugins-config-volume\") pod \"payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth\" (UID: \"80f37b1e-b948-45dc-b809-a7e6819afe8b\") " pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"
Sep 09 18:50:22.421841 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.421811    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7vjd\" (UniqueName: \"kubernetes.io/projected/80f37b1e-b948-45dc-b809-a7e6819afe8b-kube-api-access-f7vjd\") pod \"payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth\" (UID: \"80f37b1e-b948-45dc-b809-a7e6819afe8b\") " pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"
Sep 09 18:50:22.492400 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.492351    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"
Sep 09 18:50:22.655542 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.655508    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"]
Sep 09 18:50:22.658192 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:50:22.658164    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf663d651_b894_4dcd_9391_726087715800.slice/crio-71f77c13ff4a11d1295d180523cd796eda345b5ab3f57ed55047916293728148 WatchSource:0}: Error finding container 71f77c13ff4a11d1295d180523cd796eda345b5ab3f57ed55047916293728148: Status 404 returned error can't find the container with id 71f77c13ff4a11d1295d180523cd796eda345b5ab3f57ed55047916293728148
Sep 09 18:50:22.769729 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:22.769702    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"]
Sep 09 18:50:22.771760 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:50:22.771722    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f37b1e_b948_45dc_b809_a7e6819afe8b.slice/crio-3da8144065c6e6528ef1d1524d77cd8e665f2b4db371a3de9c89c33b81071c51 WatchSource:0}: Error finding container 3da8144065c6e6528ef1d1524d77cd8e665f2b4db371a3de9c89c33b81071c51: Status 404 returned error can't find the container with id 3da8144065c6e6528ef1d1524d77cd8e665f2b4db371a3de9c89c33b81071c51
Sep 09 18:50:23.453170 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:23.453134    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8" event={"ID":"f663d651-b894-4dcd-9391-726087715800","Type":"ContainerStarted","Data":"740e9524c30e10694bd37ac677441d1e9703d90f345c1b6fcc2671e58cf41336"}
Sep 09 18:50:23.453826 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:23.453191    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8" event={"ID":"f663d651-b894-4dcd-9391-726087715800","Type":"ContainerStarted","Data":"71f77c13ff4a11d1295d180523cd796eda345b5ab3f57ed55047916293728148"}
Sep 09 18:50:23.453826 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:23.453230    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"
Sep 09 18:50:23.455447 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:23.455413    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth" event={"ID":"80f37b1e-b948-45dc-b809-a7e6819afe8b","Type":"ContainerStarted","Data":"bb31467f68ebcf67f13b05d9acb5d692531cb9277e8d9c28b19dacf2939108c7"}
Sep 09 18:50:23.455613 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:23.455474    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth" event={"ID":"80f37b1e-b948-45dc-b809-a7e6819afe8b","Type":"ContainerStarted","Data":"3da8144065c6e6528ef1d1524d77cd8e665f2b4db371a3de9c89c33b81071c51"}
Sep 09 18:50:23.455613 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:23.455538    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"
Sep 09 18:50:23.474414 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:23.474357    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8" podStartSLOduration=1.474337278 podStartE2EDuration="1.474337278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:50:23.471262023 +0000 UTC m=+1111.708631830" watchObservedRunningTime="2026-09-09 18:50:23.474337278 +0000 UTC m=+1111.711707142"
Sep 09 18:50:23.492188 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:23.492133    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth" podStartSLOduration=1.492113459 podStartE2EDuration="1.492113459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:50:23.490448265 +0000 UTC m=+1111.727818087" watchObservedRunningTime="2026-09-09 18:50:23.492113459 +0000 UTC m=+1111.729483307"
Sep 09 18:50:27.856094 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:27.856052    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7" podUID="e5f0450b-b600-45c5-afc8-70d24d65d905" containerName="main" containerID="cri-o://1642cea8b719ce357b12372b2959d1c22c96c26b612af8f6718d136cc577d511" gracePeriod=45
Sep 09 18:50:28.110546 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.110522    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:50:28.273992 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.273954    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-model-cache\") pod \"e5f0450b-b600-45c5-afc8-70d24d65d905\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") "
Sep 09 18:50:28.274190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.274044    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-dshm\") pod \"e5f0450b-b600-45c5-afc8-70d24d65d905\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") "
Sep 09 18:50:28.274190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.274059    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-tmp-dir\") pod \"e5f0450b-b600-45c5-afc8-70d24d65d905\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") "
Sep 09 18:50:28.274190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.274095    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-home\") pod \"e5f0450b-b600-45c5-afc8-70d24d65d905\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") "
Sep 09 18:50:28.274190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.274150    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxfwf\" (UniqueName: \"kubernetes.io/projected/e5f0450b-b600-45c5-afc8-70d24d65d905-kube-api-access-bxfwf\") pod \"e5f0450b-b600-45c5-afc8-70d24d65d905\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") "
Sep 09 18:50:28.274190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.274182    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f0450b-b600-45c5-afc8-70d24d65d905-tls-certs\") pod \"e5f0450b-b600-45c5-afc8-70d24d65d905\" (UID: \"e5f0450b-b600-45c5-afc8-70d24d65d905\") "
Sep 09 18:50:28.274462 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.274257    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5f0450b-b600-45c5-afc8-70d24d65d905" (UID: "e5f0450b-b600-45c5-afc8-70d24d65d905"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:50:28.274462 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.274357    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5f0450b-b600-45c5-afc8-70d24d65d905" (UID: "e5f0450b-b600-45c5-afc8-70d24d65d905"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:50:28.274462 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.274390    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-home" (OuterVolumeSpecName: "home") pod "e5f0450b-b600-45c5-afc8-70d24d65d905" (UID: "e5f0450b-b600-45c5-afc8-70d24d65d905"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:50:28.274618 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.274554    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:28.274618 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.274568    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:28.274618 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.274577    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:28.276354 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.276331    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f0450b-b600-45c5-afc8-70d24d65d905-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5f0450b-b600-45c5-afc8-70d24d65d905" (UID: "e5f0450b-b600-45c5-afc8-70d24d65d905"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:50:28.276354 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.276340    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-dshm" (OuterVolumeSpecName: "dshm") pod "e5f0450b-b600-45c5-afc8-70d24d65d905" (UID: "e5f0450b-b600-45c5-afc8-70d24d65d905"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:50:28.276543 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.276379    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f0450b-b600-45c5-afc8-70d24d65d905-kube-api-access-bxfwf" (OuterVolumeSpecName: "kube-api-access-bxfwf") pod "e5f0450b-b600-45c5-afc8-70d24d65d905" (UID: "e5f0450b-b600-45c5-afc8-70d24d65d905"). InnerVolumeSpecName "kube-api-access-bxfwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:50:28.375922 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.375818    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5f0450b-b600-45c5-afc8-70d24d65d905-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:28.375922 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.375860    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxfwf\" (UniqueName: \"kubernetes.io/projected/e5f0450b-b600-45c5-afc8-70d24d65d905-kube-api-access-bxfwf\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:28.375922 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.375871    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5f0450b-b600-45c5-afc8-70d24d65d905-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:28.485040 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.484999    2575 generic.go:356] "Generic (PLEG): container finished" podID="e5f0450b-b600-45c5-afc8-70d24d65d905" containerID="1642cea8b719ce357b12372b2959d1c22c96c26b612af8f6718d136cc577d511" exitCode=0
Sep 09 18:50:28.485238 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.485045    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"
Sep 09 18:50:28.485238 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.485075    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7" event={"ID":"e5f0450b-b600-45c5-afc8-70d24d65d905","Type":"ContainerDied","Data":"1642cea8b719ce357b12372b2959d1c22c96c26b612af8f6718d136cc577d511"}
Sep 09 18:50:28.485238 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.485119    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7" event={"ID":"e5f0450b-b600-45c5-afc8-70d24d65d905","Type":"ContainerDied","Data":"894abc4305ba2224672338655834fa180b5450acce90dd5f77290719aee245c4"}
Sep 09 18:50:28.485238 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.485136    2575 scope.go:117] "RemoveContainer" containerID="1642cea8b719ce357b12372b2959d1c22c96c26b612af8f6718d136cc577d511"
Sep 09 18:50:28.494816 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.494796    2575 scope.go:117] "RemoveContainer" containerID="1642cea8b719ce357b12372b2959d1c22c96c26b612af8f6718d136cc577d511"
Sep 09 18:50:28.495133 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:50:28.495109    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1642cea8b719ce357b12372b2959d1c22c96c26b612af8f6718d136cc577d511\": container with ID starting with 1642cea8b719ce357b12372b2959d1c22c96c26b612af8f6718d136cc577d511 not found: ID does not exist" containerID="1642cea8b719ce357b12372b2959d1c22c96c26b612af8f6718d136cc577d511"
Sep 09 18:50:28.495185 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.495144    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1642cea8b719ce357b12372b2959d1c22c96c26b612af8f6718d136cc577d511"} err="failed to get container status \"1642cea8b719ce357b12372b2959d1c22c96c26b612af8f6718d136cc577d511\": rpc error: code = NotFound desc = could not find container \"1642cea8b719ce357b12372b2959d1c22c96c26b612af8f6718d136cc577d511\": container with ID starting with 1642cea8b719ce357b12372b2959d1c22c96c26b612af8f6718d136cc577d511 not found: ID does not exist"
Sep 09 18:50:28.515154 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.515117    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"]
Sep 09 18:50:28.518115 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:28.518083    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-8b6cce27-kserve-587f66647-8gzh7"]
Sep 09 18:50:30.321747 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:30.321714    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5f0450b-b600-45c5-afc8-70d24d65d905" path="/var/lib/kubelet/pods/e5f0450b-b600-45c5-afc8-70d24d65d905/volumes"
Sep 09 18:50:30.433889 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:30.433857    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"
Sep 09 18:50:30.434059 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:30.433934    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"
Sep 09 18:50:34.463447 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:34.463413    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"
Sep 09 18:50:34.463965 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:34.463510    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"
Sep 09 18:50:35.044201 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.044169    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"]
Sep 09 18:50:35.044672 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.044656    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5f0450b-b600-45c5-afc8-70d24d65d905" containerName="main"
Sep 09 18:50:35.044732 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.044673    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5f0450b-b600-45c5-afc8-70d24d65d905" containerName="main"
Sep 09 18:50:35.044771 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.044742    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5f0450b-b600-45c5-afc8-70d24d65d905" containerName="main"
Sep 09 18:50:35.049216 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.049198    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.051973 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.051939    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-77aadf\"/\"openshift-service-ca.crt\""
Sep 09 18:50:35.052995 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.052961    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-77aadf\"/\"kube-root-ca.crt\""
Sep 09 18:50:35.053142 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.053063    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-77aadf\"/\"test-model-77aadf-kserve-self-signed-certs\""
Sep 09 18:50:35.053383 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.053356    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-77aadf\"/\"default-dockercfg-5q9pg\""
Sep 09 18:50:35.056298 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.056261    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"]
Sep 09 18:50:35.143066 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.143015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-home\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.143066 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.143069    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-model-cache\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.143297 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.143134    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b850a27c-5abe-44d9-af36-99752811e17e-tls-certs\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.143297 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.143166    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-dshm\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.143297 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.143218    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-tmp-dir\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.143297 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.143287    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wth9q\" (UniqueName: \"kubernetes.io/projected/b850a27c-5abe-44d9-af36-99752811e17e-kube-api-access-wth9q\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.244791 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.244753    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-tmp-dir\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.244962 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.244919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wth9q\" (UniqueName: \"kubernetes.io/projected/b850a27c-5abe-44d9-af36-99752811e17e-kube-api-access-wth9q\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.245053 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.245031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-home\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.245090 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.245077    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-model-cache\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.245132 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.245116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b850a27c-5abe-44d9-af36-99752811e17e-tls-certs\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.245201 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.245172    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-dshm\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.245257 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.245198    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-tmp-dir\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.245413 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.245391    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-home\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.245486 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.245416    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-model-cache\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.247387 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.247365    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-dshm\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.247680 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.247663    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b850a27c-5abe-44d9-af36-99752811e17e-tls-certs\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.253069 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.253046    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wth9q\" (UniqueName: \"kubernetes.io/projected/b850a27c-5abe-44d9-af36-99752811e17e-kube-api-access-wth9q\") pod \"test-model-77aadf-kserve-d99d89769-q5t2p\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") " pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.360651 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.360609    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:35.639751 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:35.639719    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"]
Sep 09 18:50:35.641850 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:50:35.641822    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb850a27c_5abe_44d9_af36_99752811e17e.slice/crio-6eef14dc879a5b627d16c57d2cd5ff74e93adf1cb2089e68f0f7fcea816d6280 WatchSource:0}: Error finding container 6eef14dc879a5b627d16c57d2cd5ff74e93adf1cb2089e68f0f7fcea816d6280: Status 404 returned error can't find the container with id 6eef14dc879a5b627d16c57d2cd5ff74e93adf1cb2089e68f0f7fcea816d6280
Sep 09 18:50:36.539869 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:36.539832    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p" event={"ID":"b850a27c-5abe-44d9-af36-99752811e17e","Type":"ContainerStarted","Data":"1a2d238f0676eeb66e899f4ab4fd6bedf55b6f0bb58c63034193884ec3c64f12"}
Sep 09 18:50:36.539869 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:36.539878    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p" event={"ID":"b850a27c-5abe-44d9-af36-99752811e17e","Type":"ContainerStarted","Data":"6eef14dc879a5b627d16c57d2cd5ff74e93adf1cb2089e68f0f7fcea816d6280"}
Sep 09 18:50:36.561316 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:36.561262    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p" podStartSLOduration=1.561242598 podStartE2EDuration="1.561242598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:50:36.558762152 +0000 UTC m=+1124.796131967" watchObservedRunningTime="2026-09-09 18:50:36.561242598 +0000 UTC m=+1124.798612413"
Sep 09 18:50:45.361681 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:45.361639    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:45.361681 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:45.361691    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:45.374800 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:45.374765    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:45.599278 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:45.599248    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:50:51.854739 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:51.854700    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"]
Sep 09 18:50:51.855225 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:51.855024    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth" podUID="80f37b1e-b948-45dc-b809-a7e6819afe8b" containerName="payload-processing" containerID="cri-o://bb31467f68ebcf67f13b05d9acb5d692531cb9277e8d9c28b19dacf2939108c7" gracePeriod=30
Sep 09 18:50:51.871751 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:51.871722    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"]
Sep 09 18:50:51.871964 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:51.871940    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8" podUID="f663d651-b894-4dcd-9391-726087715800" containerName="payload-pre-processing" containerID="cri-o://740e9524c30e10694bd37ac677441d1e9703d90f345c1b6fcc2671e58cf41336" gracePeriod=30
Sep 09 18:50:52.142986 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.142954    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"
Sep 09 18:50:52.167038 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.167012    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"
Sep 09 18:50:52.232290 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.232250    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7vjd\" (UniqueName: \"kubernetes.io/projected/80f37b1e-b948-45dc-b809-a7e6819afe8b-kube-api-access-f7vjd\") pod \"80f37b1e-b948-45dc-b809-a7e6819afe8b\" (UID: \"80f37b1e-b948-45dc-b809-a7e6819afe8b\") "
Sep 09 18:50:52.232492 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.232308    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rsrl\" (UniqueName: \"kubernetes.io/projected/f663d651-b894-4dcd-9391-726087715800-kube-api-access-9rsrl\") pod \"f663d651-b894-4dcd-9391-726087715800\" (UID: \"f663d651-b894-4dcd-9391-726087715800\") "
Sep 09 18:50:52.232492 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.232377    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f37b1e-b948-45dc-b809-a7e6819afe8b-plugins-config-volume\") pod \"80f37b1e-b948-45dc-b809-a7e6819afe8b\" (UID: \"80f37b1e-b948-45dc-b809-a7e6819afe8b\") "
Sep 09 18:50:52.232492 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.232455    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f663d651-b894-4dcd-9391-726087715800-plugins-config-volume\") pod \"f663d651-b894-4dcd-9391-726087715800\" (UID: \"f663d651-b894-4dcd-9391-726087715800\") "
Sep 09 18:50:52.232866 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.232834    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f37b1e-b948-45dc-b809-a7e6819afe8b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80f37b1e-b948-45dc-b809-a7e6819afe8b" (UID: "80f37b1e-b948-45dc-b809-a7e6819afe8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:50:52.232948 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.232841    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f663d651-b894-4dcd-9391-726087715800-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f663d651-b894-4dcd-9391-726087715800" (UID: "f663d651-b894-4dcd-9391-726087715800"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:50:52.233028 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.233008    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f37b1e-b948-45dc-b809-a7e6819afe8b-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:52.233084 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.233032    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f663d651-b894-4dcd-9391-726087715800-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:52.235090 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.235060    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f37b1e-b948-45dc-b809-a7e6819afe8b-kube-api-access-f7vjd" (OuterVolumeSpecName: "kube-api-access-f7vjd") pod "80f37b1e-b948-45dc-b809-a7e6819afe8b" (UID: "80f37b1e-b948-45dc-b809-a7e6819afe8b"). InnerVolumeSpecName "kube-api-access-f7vjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:50:52.235233 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.235213    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f663d651-b894-4dcd-9391-726087715800-kube-api-access-9rsrl" (OuterVolumeSpecName: "kube-api-access-9rsrl") pod "f663d651-b894-4dcd-9391-726087715800" (UID: "f663d651-b894-4dcd-9391-726087715800"). InnerVolumeSpecName "kube-api-access-9rsrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:50:52.334338 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.334194    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7vjd\" (UniqueName: \"kubernetes.io/projected/80f37b1e-b948-45dc-b809-a7e6819afe8b-kube-api-access-f7vjd\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:52.334338 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.334226    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rsrl\" (UniqueName: \"kubernetes.io/projected/f663d651-b894-4dcd-9391-726087715800-kube-api-access-9rsrl\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:50:52.635395 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.635299    2575 generic.go:356] "Generic (PLEG): container finished" podID="80f37b1e-b948-45dc-b809-a7e6819afe8b" containerID="bb31467f68ebcf67f13b05d9acb5d692531cb9277e8d9c28b19dacf2939108c7" exitCode=0
Sep 09 18:50:52.635395 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.635343    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"
Sep 09 18:50:52.635673 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.635383    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth" event={"ID":"80f37b1e-b948-45dc-b809-a7e6819afe8b","Type":"ContainerDied","Data":"bb31467f68ebcf67f13b05d9acb5d692531cb9277e8d9c28b19dacf2939108c7"}
Sep 09 18:50:52.635673 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.635443    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth" event={"ID":"80f37b1e-b948-45dc-b809-a7e6819afe8b","Type":"ContainerDied","Data":"3da8144065c6e6528ef1d1524d77cd8e665f2b4db371a3de9c89c33b81071c51"}
Sep 09 18:50:52.635673 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.635473    2575 scope.go:117] "RemoveContainer" containerID="bb31467f68ebcf67f13b05d9acb5d692531cb9277e8d9c28b19dacf2939108c7"
Sep 09 18:50:52.636939 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.636905    2575 generic.go:356] "Generic (PLEG): container finished" podID="f663d651-b894-4dcd-9391-726087715800" containerID="740e9524c30e10694bd37ac677441d1e9703d90f345c1b6fcc2671e58cf41336" exitCode=0
Sep 09 18:50:52.637048 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.636948    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"
Sep 09 18:50:52.637048 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.636988    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8" event={"ID":"f663d651-b894-4dcd-9391-726087715800","Type":"ContainerDied","Data":"740e9524c30e10694bd37ac677441d1e9703d90f345c1b6fcc2671e58cf41336"}
Sep 09 18:50:52.637048 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.637032    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8" event={"ID":"f663d651-b894-4dcd-9391-726087715800","Type":"ContainerDied","Data":"71f77c13ff4a11d1295d180523cd796eda345b5ab3f57ed55047916293728148"}
Sep 09 18:50:52.647074 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.646996    2575 scope.go:117] "RemoveContainer" containerID="bb31467f68ebcf67f13b05d9acb5d692531cb9277e8d9c28b19dacf2939108c7"
Sep 09 18:50:52.647551 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:50:52.647526    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb31467f68ebcf67f13b05d9acb5d692531cb9277e8d9c28b19dacf2939108c7\": container with ID starting with bb31467f68ebcf67f13b05d9acb5d692531cb9277e8d9c28b19dacf2939108c7 not found: ID does not exist" containerID="bb31467f68ebcf67f13b05d9acb5d692531cb9277e8d9c28b19dacf2939108c7"
Sep 09 18:50:52.647650 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.647559    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb31467f68ebcf67f13b05d9acb5d692531cb9277e8d9c28b19dacf2939108c7"} err="failed to get container status \"bb31467f68ebcf67f13b05d9acb5d692531cb9277e8d9c28b19dacf2939108c7\": rpc error: code = NotFound desc = could not find container \"bb31467f68ebcf67f13b05d9acb5d692531cb9277e8d9c28b19dacf2939108c7\": container with ID starting with bb31467f68ebcf67f13b05d9acb5d692531cb9277e8d9c28b19dacf2939108c7 not found: ID does not exist"
Sep 09 18:50:52.647650 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.647577    2575 scope.go:117] "RemoveContainer" containerID="740e9524c30e10694bd37ac677441d1e9703d90f345c1b6fcc2671e58cf41336"
Sep 09 18:50:52.656249 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.656230    2575 scope.go:117] "RemoveContainer" containerID="740e9524c30e10694bd37ac677441d1e9703d90f345c1b6fcc2671e58cf41336"
Sep 09 18:50:52.656527 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:50:52.656495    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"740e9524c30e10694bd37ac677441d1e9703d90f345c1b6fcc2671e58cf41336\": container with ID starting with 740e9524c30e10694bd37ac677441d1e9703d90f345c1b6fcc2671e58cf41336 not found: ID does not exist" containerID="740e9524c30e10694bd37ac677441d1e9703d90f345c1b6fcc2671e58cf41336"
Sep 09 18:50:52.656573 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.656535    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740e9524c30e10694bd37ac677441d1e9703d90f345c1b6fcc2671e58cf41336"} err="failed to get container status \"740e9524c30e10694bd37ac677441d1e9703d90f345c1b6fcc2671e58cf41336\": rpc error: code = NotFound desc = could not find container \"740e9524c30e10694bd37ac677441d1e9703d90f345c1b6fcc2671e58cf41336\": container with ID starting with 740e9524c30e10694bd37ac677441d1e9703d90f345c1b6fcc2671e58cf41336 not found: ID does not exist"
Sep 09 18:50:52.680767 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.680731    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"]
Sep 09 18:50:52.685922 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.685893    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-edd73ecc-7bf7cd4b5d-6pcr8"]
Sep 09 18:50:52.704617 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.704579    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"]
Sep 09 18:50:52.712036 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:52.712003    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-edd73ecc-57f476f7fc-qrnth"]
Sep 09 18:50:54.321037 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:54.321005    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f37b1e-b948-45dc-b809-a7e6819afe8b" path="/var/lib/kubelet/pods/80f37b1e-b948-45dc-b809-a7e6819afe8b/volumes"
Sep 09 18:50:54.321436 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:50:54.321396    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f663d651-b894-4dcd-9391-726087715800" path="/var/lib/kubelet/pods/f663d651-b894-4dcd-9391-726087715800/volumes"
Sep 09 18:51:07.237068 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.237030    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"]
Sep 09 18:51:07.237727 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.237705    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f37b1e-b948-45dc-b809-a7e6819afe8b" containerName="payload-processing"
Sep 09 18:51:07.237817 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.237729    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f37b1e-b948-45dc-b809-a7e6819afe8b" containerName="payload-processing"
Sep 09 18:51:07.237817 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.237746    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f663d651-b894-4dcd-9391-726087715800" containerName="payload-pre-processing"
Sep 09 18:51:07.237817 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.237754    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f663d651-b894-4dcd-9391-726087715800" containerName="payload-pre-processing"
Sep 09 18:51:07.237970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.237889    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f37b1e-b948-45dc-b809-a7e6819afe8b" containerName="payload-processing"
Sep 09 18:51:07.237970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.237904    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f663d651-b894-4dcd-9391-726087715800" containerName="payload-pre-processing"
Sep 09 18:51:07.242886 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.242862    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.245803 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.245777    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f8eac9\"/\"explicit-model-f8eac9-kserve-self-signed-certs\""
Sep 09 18:51:07.247315 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.247285    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f8eac9\"/\"kube-root-ca.crt\""
Sep 09 18:51:07.247468 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.247415    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f8eac9\"/\"default-dockercfg-b2b8s\""
Sep 09 18:51:07.247560 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.247509    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f8eac9\"/\"openshift-service-ca.crt\""
Sep 09 18:51:07.259882 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.259851    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"]
Sep 09 18:51:07.395428 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.395234    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-dshm\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.395687 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.395594    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b05e05-0b87-4cb1-8732-6771dc3b75d2-tls-certs\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.395687 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.395677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-model-cache\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.395828 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.395726    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-tmp-dir\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.395889 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.395867    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68kp6\" (UniqueName: \"kubernetes.io/projected/81b05e05-0b87-4cb1-8732-6771dc3b75d2-kube-api-access-68kp6\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.395945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.395912    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-home\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.497132 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.497011    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-home\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.497132 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.497099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-dshm\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.497132 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.497129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b05e05-0b87-4cb1-8732-6771dc3b75d2-tls-certs\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.497373 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.497162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-model-cache\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.497373 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.497193    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-tmp-dir\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.497373 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.497243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68kp6\" (UniqueName: \"kubernetes.io/projected/81b05e05-0b87-4cb1-8732-6771dc3b75d2-kube-api-access-68kp6\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.497594 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.497566    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-home\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.497675 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.497593    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-model-cache\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.497728 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.497686    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-tmp-dir\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.499647 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.499619    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-dshm\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.499889 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.499867    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b05e05-0b87-4cb1-8732-6771dc3b75d2-tls-certs\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.507158 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.507131    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68kp6\" (UniqueName: \"kubernetes.io/projected/81b05e05-0b87-4cb1-8732-6771dc3b75d2-kube-api-access-68kp6\") pod \"explicit-model-f8eac9-kserve-546fcbf8f-zhf7j\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") " pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.553558 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.553483    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:07.839285 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:07.839256    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"]
Sep 09 18:51:07.841118 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:51:07.841090    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81b05e05_0b87_4cb1_8732_6771dc3b75d2.slice/crio-d86d68f4a6dc36a9213d2e1e5b71e4bdd510c1e00a7855c1eab462a4fcdf36cf WatchSource:0}: Error finding container d86d68f4a6dc36a9213d2e1e5b71e4bdd510c1e00a7855c1eab462a4fcdf36cf: Status 404 returned error can't find the container with id d86d68f4a6dc36a9213d2e1e5b71e4bdd510c1e00a7855c1eab462a4fcdf36cf
Sep 09 18:51:08.730746 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:08.730704    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j" event={"ID":"81b05e05-0b87-4cb1-8732-6771dc3b75d2","Type":"ContainerStarted","Data":"03d8adec6f6367028fc896fa73c2d71fe8fe76fe5cb80fc54f5cef63ec231408"}
Sep 09 18:51:08.730746 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:08.730757    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j" event={"ID":"81b05e05-0b87-4cb1-8732-6771dc3b75d2","Type":"ContainerStarted","Data":"d86d68f4a6dc36a9213d2e1e5b71e4bdd510c1e00a7855c1eab462a4fcdf36cf"}
Sep 09 18:51:08.753744 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:08.753690    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j" podStartSLOduration=1.7536738710000002 podStartE2EDuration="1.753673871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:51:08.751808256 +0000 UTC m=+1156.989178071" watchObservedRunningTime="2026-09-09 18:51:08.753673871 +0000 UTC m=+1156.991043695"
Sep 09 18:51:10.033403 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:10.033365    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"]
Sep 09 18:51:17.554227 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:17.554189    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:20.753045 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:20.753001    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"]
Sep 09 18:51:20.753523 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:20.753248    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b" podUID="67634d09-6a9c-4182-83a1-f49b799201bc" containerName="payload-processing" containerID="cri-o://cb6756ffe7c45438d3b7fdcaba8295e5acbd50f5e4ab81c7a6fba95954b5dbe9" gracePeriod=30
Sep 09 18:51:20.765115 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:20.765076    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"]
Sep 09 18:51:20.765388 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:20.765362    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd" podUID="af3eb9b8-6844-43b0-be35-f9a64e4a8218" containerName="payload-pre-processing" containerID="cri-o://792c65200d8bbf0c0927b2305b5a0d6a35e58942a10e417b21b40cb40f02fe5f" gracePeriod=30
Sep 09 18:51:21.050100 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.050067    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"
Sep 09 18:51:21.053081 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.053051    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"
Sep 09 18:51:21.127847 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.127811    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67634d09-6a9c-4182-83a1-f49b799201bc-plugins-config-volume\") pod \"67634d09-6a9c-4182-83a1-f49b799201bc\" (UID: \"67634d09-6a9c-4182-83a1-f49b799201bc\") "
Sep 09 18:51:21.128076 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.127930    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kdfj\" (UniqueName: \"kubernetes.io/projected/af3eb9b8-6844-43b0-be35-f9a64e4a8218-kube-api-access-7kdfj\") pod \"af3eb9b8-6844-43b0-be35-f9a64e4a8218\" (UID: \"af3eb9b8-6844-43b0-be35-f9a64e4a8218\") "
Sep 09 18:51:21.128076 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.127949    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnh6b\" (UniqueName: \"kubernetes.io/projected/67634d09-6a9c-4182-83a1-f49b799201bc-kube-api-access-jnh6b\") pod \"67634d09-6a9c-4182-83a1-f49b799201bc\" (UID: \"67634d09-6a9c-4182-83a1-f49b799201bc\") "
Sep 09 18:51:21.128076 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.127970    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3eb9b8-6844-43b0-be35-f9a64e4a8218-plugins-config-volume\") pod \"af3eb9b8-6844-43b0-be35-f9a64e4a8218\" (UID: \"af3eb9b8-6844-43b0-be35-f9a64e4a8218\") "
Sep 09 18:51:21.128404 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.128373    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67634d09-6a9c-4182-83a1-f49b799201bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67634d09-6a9c-4182-83a1-f49b799201bc" (UID: "67634d09-6a9c-4182-83a1-f49b799201bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:51:21.128469 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.128411    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3eb9b8-6844-43b0-be35-f9a64e4a8218-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af3eb9b8-6844-43b0-be35-f9a64e4a8218" (UID: "af3eb9b8-6844-43b0-be35-f9a64e4a8218"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:51:21.130376 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.130347    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67634d09-6a9c-4182-83a1-f49b799201bc-kube-api-access-jnh6b" (OuterVolumeSpecName: "kube-api-access-jnh6b") pod "67634d09-6a9c-4182-83a1-f49b799201bc" (UID: "67634d09-6a9c-4182-83a1-f49b799201bc"). InnerVolumeSpecName "kube-api-access-jnh6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:51:21.130630 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.130374    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3eb9b8-6844-43b0-be35-f9a64e4a8218-kube-api-access-7kdfj" (OuterVolumeSpecName: "kube-api-access-7kdfj") pod "af3eb9b8-6844-43b0-be35-f9a64e4a8218" (UID: "af3eb9b8-6844-43b0-be35-f9a64e4a8218"). InnerVolumeSpecName "kube-api-access-7kdfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:51:21.229730 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.229683    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67634d09-6a9c-4182-83a1-f49b799201bc-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:51:21.229730 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.229715    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kdfj\" (UniqueName: \"kubernetes.io/projected/af3eb9b8-6844-43b0-be35-f9a64e4a8218-kube-api-access-7kdfj\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:51:21.229730 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.229724    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnh6b\" (UniqueName: \"kubernetes.io/projected/67634d09-6a9c-4182-83a1-f49b799201bc-kube-api-access-jnh6b\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:51:21.229730 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.229734    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3eb9b8-6844-43b0-be35-f9a64e4a8218-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:51:21.812556 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.812490    2575 generic.go:356] "Generic (PLEG): container finished" podID="af3eb9b8-6844-43b0-be35-f9a64e4a8218" containerID="792c65200d8bbf0c0927b2305b5a0d6a35e58942a10e417b21b40cb40f02fe5f" exitCode=0
Sep 09 18:51:21.812556 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.812548    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"
Sep 09 18:51:21.813087 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.812589    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd" event={"ID":"af3eb9b8-6844-43b0-be35-f9a64e4a8218","Type":"ContainerDied","Data":"792c65200d8bbf0c0927b2305b5a0d6a35e58942a10e417b21b40cb40f02fe5f"}
Sep 09 18:51:21.813087 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.812647    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd" event={"ID":"af3eb9b8-6844-43b0-be35-f9a64e4a8218","Type":"ContainerDied","Data":"4b56f6120298103df083cd31e9e980f885672afcf5b17191be27eb031a6a31d4"}
Sep 09 18:51:21.813087 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.812675    2575 scope.go:117] "RemoveContainer" containerID="792c65200d8bbf0c0927b2305b5a0d6a35e58942a10e417b21b40cb40f02fe5f"
Sep 09 18:51:21.814102 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.814078    2575 generic.go:356] "Generic (PLEG): container finished" podID="67634d09-6a9c-4182-83a1-f49b799201bc" containerID="cb6756ffe7c45438d3b7fdcaba8295e5acbd50f5e4ab81c7a6fba95954b5dbe9" exitCode=0
Sep 09 18:51:21.814167 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.814117    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"
Sep 09 18:51:21.814167 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.814150    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b" event={"ID":"67634d09-6a9c-4182-83a1-f49b799201bc","Type":"ContainerDied","Data":"cb6756ffe7c45438d3b7fdcaba8295e5acbd50f5e4ab81c7a6fba95954b5dbe9"}
Sep 09 18:51:21.814248 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.814205    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b" event={"ID":"67634d09-6a9c-4182-83a1-f49b799201bc","Type":"ContainerDied","Data":"ec8ccf5ed042634b2d27895f4a3b8cebe94eb94b7fd1a305b5bba474d6f52370"}
Sep 09 18:51:21.822680 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.822649    2575 scope.go:117] "RemoveContainer" containerID="792c65200d8bbf0c0927b2305b5a0d6a35e58942a10e417b21b40cb40f02fe5f"
Sep 09 18:51:21.823033 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:51:21.823008    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"792c65200d8bbf0c0927b2305b5a0d6a35e58942a10e417b21b40cb40f02fe5f\": container with ID starting with 792c65200d8bbf0c0927b2305b5a0d6a35e58942a10e417b21b40cb40f02fe5f not found: ID does not exist" containerID="792c65200d8bbf0c0927b2305b5a0d6a35e58942a10e417b21b40cb40f02fe5f"
Sep 09 18:51:21.823131 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.823041    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792c65200d8bbf0c0927b2305b5a0d6a35e58942a10e417b21b40cb40f02fe5f"} err="failed to get container status \"792c65200d8bbf0c0927b2305b5a0d6a35e58942a10e417b21b40cb40f02fe5f\": rpc error: code = NotFound desc = could not find container \"792c65200d8bbf0c0927b2305b5a0d6a35e58942a10e417b21b40cb40f02fe5f\": container with ID starting with 792c65200d8bbf0c0927b2305b5a0d6a35e58942a10e417b21b40cb40f02fe5f not found: ID does not exist"
Sep 09 18:51:21.823131 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.823058    2575 scope.go:117] "RemoveContainer" containerID="cb6756ffe7c45438d3b7fdcaba8295e5acbd50f5e4ab81c7a6fba95954b5dbe9"
Sep 09 18:51:21.832427 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.832404    2575 scope.go:117] "RemoveContainer" containerID="cb6756ffe7c45438d3b7fdcaba8295e5acbd50f5e4ab81c7a6fba95954b5dbe9"
Sep 09 18:51:21.832808 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:51:21.832786    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb6756ffe7c45438d3b7fdcaba8295e5acbd50f5e4ab81c7a6fba95954b5dbe9\": container with ID starting with cb6756ffe7c45438d3b7fdcaba8295e5acbd50f5e4ab81c7a6fba95954b5dbe9 not found: ID does not exist" containerID="cb6756ffe7c45438d3b7fdcaba8295e5acbd50f5e4ab81c7a6fba95954b5dbe9"
Sep 09 18:51:21.832899 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.832817    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb6756ffe7c45438d3b7fdcaba8295e5acbd50f5e4ab81c7a6fba95954b5dbe9"} err="failed to get container status \"cb6756ffe7c45438d3b7fdcaba8295e5acbd50f5e4ab81c7a6fba95954b5dbe9\": rpc error: code = NotFound desc = could not find container \"cb6756ffe7c45438d3b7fdcaba8295e5acbd50f5e4ab81c7a6fba95954b5dbe9\": container with ID starting with cb6756ffe7c45438d3b7fdcaba8295e5acbd50f5e4ab81c7a6fba95954b5dbe9 not found: ID does not exist"
Sep 09 18:51:21.863089 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.863053    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"]
Sep 09 18:51:21.866430 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.866399    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f8eac9-5c579865c-8pk5b"]
Sep 09 18:51:21.901253 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.901216    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"]
Sep 09 18:51:21.919667 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:21.919635    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f8eac9-6f84965974-8fvhd"]
Sep 09 18:51:22.321039 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:22.321004    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67634d09-6a9c-4182-83a1-f49b799201bc" path="/var/lib/kubelet/pods/67634d09-6a9c-4182-83a1-f49b799201bc/volumes"
Sep 09 18:51:22.321418 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:22.321404    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af3eb9b8-6844-43b0-be35-f9a64e4a8218" path="/var/lib/kubelet/pods/af3eb9b8-6844-43b0-be35-f9a64e4a8218/volumes"
Sep 09 18:51:25.765899 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:25.765840    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j" podUID="81b05e05-0b87-4cb1-8732-6771dc3b75d2" containerName="main" containerID="cri-o://03d8adec6f6367028fc896fa73c2d71fe8fe76fe5cb80fc54f5cef63ec231408" gracePeriod=45
Sep 09 18:51:26.041633 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.041603    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:26.083065 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.083026    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68kp6\" (UniqueName: \"kubernetes.io/projected/81b05e05-0b87-4cb1-8732-6771dc3b75d2-kube-api-access-68kp6\") pod \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") "
Sep 09 18:51:26.083272 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.083098    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-model-cache\") pod \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") "
Sep 09 18:51:26.083272 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.083136    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-dshm\") pod \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") "
Sep 09 18:51:26.083272 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.083162    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-home\") pod \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") "
Sep 09 18:51:26.083520 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.083412    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "81b05e05-0b87-4cb1-8732-6771dc3b75d2" (UID: "81b05e05-0b87-4cb1-8732-6771dc3b75d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:51:26.083592 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.083546    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-home" (OuterVolumeSpecName: "home") pod "81b05e05-0b87-4cb1-8732-6771dc3b75d2" (UID: "81b05e05-0b87-4cb1-8732-6771dc3b75d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:51:26.083592 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.083239    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-tmp-dir\") pod \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") "
Sep 09 18:51:26.083688 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.083674    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b05e05-0b87-4cb1-8732-6771dc3b75d2-tls-certs\") pod \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\" (UID: \"81b05e05-0b87-4cb1-8732-6771dc3b75d2\") "
Sep 09 18:51:26.083735 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.083697    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81b05e05-0b87-4cb1-8732-6771dc3b75d2" (UID: "81b05e05-0b87-4cb1-8732-6771dc3b75d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:51:26.084179 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.084115    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:51:26.084179 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.084141    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:51:26.084179 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.084155    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:51:26.085599 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.085570    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b05e05-0b87-4cb1-8732-6771dc3b75d2-kube-api-access-68kp6" (OuterVolumeSpecName: "kube-api-access-68kp6") pod "81b05e05-0b87-4cb1-8732-6771dc3b75d2" (UID: "81b05e05-0b87-4cb1-8732-6771dc3b75d2"). InnerVolumeSpecName "kube-api-access-68kp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:51:26.085718 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.085684    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b05e05-0b87-4cb1-8732-6771dc3b75d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81b05e05-0b87-4cb1-8732-6771dc3b75d2" (UID: "81b05e05-0b87-4cb1-8732-6771dc3b75d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:51:26.085805 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.085783    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-dshm" (OuterVolumeSpecName: "dshm") pod "81b05e05-0b87-4cb1-8732-6771dc3b75d2" (UID: "81b05e05-0b87-4cb1-8732-6771dc3b75d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:51:26.185652 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.185604    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68kp6\" (UniqueName: \"kubernetes.io/projected/81b05e05-0b87-4cb1-8732-6771dc3b75d2-kube-api-access-68kp6\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:51:26.185652 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.185645    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b05e05-0b87-4cb1-8732-6771dc3b75d2-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:51:26.185652 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.185658    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b05e05-0b87-4cb1-8732-6771dc3b75d2-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:51:26.845433 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.845397    2575 generic.go:356] "Generic (PLEG): container finished" podID="81b05e05-0b87-4cb1-8732-6771dc3b75d2" containerID="03d8adec6f6367028fc896fa73c2d71fe8fe76fe5cb80fc54f5cef63ec231408" exitCode=0
Sep 09 18:51:26.845906 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.845440    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"
Sep 09 18:51:26.845906 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.845492    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j" event={"ID":"81b05e05-0b87-4cb1-8732-6771dc3b75d2","Type":"ContainerDied","Data":"03d8adec6f6367028fc896fa73c2d71fe8fe76fe5cb80fc54f5cef63ec231408"}
Sep 09 18:51:26.845906 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.845556    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j" event={"ID":"81b05e05-0b87-4cb1-8732-6771dc3b75d2","Type":"ContainerDied","Data":"d86d68f4a6dc36a9213d2e1e5b71e4bdd510c1e00a7855c1eab462a4fcdf36cf"}
Sep 09 18:51:26.845906 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.845575    2575 scope.go:117] "RemoveContainer" containerID="03d8adec6f6367028fc896fa73c2d71fe8fe76fe5cb80fc54f5cef63ec231408"
Sep 09 18:51:26.854288 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.854260    2575 scope.go:117] "RemoveContainer" containerID="03d8adec6f6367028fc896fa73c2d71fe8fe76fe5cb80fc54f5cef63ec231408"
Sep 09 18:51:26.854620 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:51:26.854600    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d8adec6f6367028fc896fa73c2d71fe8fe76fe5cb80fc54f5cef63ec231408\": container with ID starting with 03d8adec6f6367028fc896fa73c2d71fe8fe76fe5cb80fc54f5cef63ec231408 not found: ID does not exist" containerID="03d8adec6f6367028fc896fa73c2d71fe8fe76fe5cb80fc54f5cef63ec231408"
Sep 09 18:51:26.854676 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.854630    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d8adec6f6367028fc896fa73c2d71fe8fe76fe5cb80fc54f5cef63ec231408"} err="failed to get container status \"03d8adec6f6367028fc896fa73c2d71fe8fe76fe5cb80fc54f5cef63ec231408\": rpc error: code = NotFound desc = could not find container \"03d8adec6f6367028fc896fa73c2d71fe8fe76fe5cb80fc54f5cef63ec231408\": container with ID starting with 03d8adec6f6367028fc896fa73c2d71fe8fe76fe5cb80fc54f5cef63ec231408 not found: ID does not exist"
Sep 09 18:51:26.878842 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.878803    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"]
Sep 09 18:51:26.881708 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:26.881661    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-f8eac9/explicit-model-f8eac9-kserve-546fcbf8f-zhf7j"]
Sep 09 18:51:28.322017 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:28.321975    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81b05e05-0b87-4cb1-8732-6771dc3b75d2" path="/var/lib/kubelet/pods/81b05e05-0b87-4cb1-8732-6771dc3b75d2/volumes"
Sep 09 18:51:42.247036 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.247000    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"]
Sep 09 18:51:42.247719 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.247700    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b05e05-0b87-4cb1-8732-6771dc3b75d2" containerName="main"
Sep 09 18:51:42.247772 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.247723    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b05e05-0b87-4cb1-8732-6771dc3b75d2" containerName="main"
Sep 09 18:51:42.247772 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.247745    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af3eb9b8-6844-43b0-be35-f9a64e4a8218" containerName="payload-pre-processing"
Sep 09 18:51:42.247772 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.247754    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="af3eb9b8-6844-43b0-be35-f9a64e4a8218" containerName="payload-pre-processing"
Sep 09 18:51:42.247876 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.247773    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67634d09-6a9c-4182-83a1-f49b799201bc" containerName="payload-processing"
Sep 09 18:51:42.247876 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.247782    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="67634d09-6a9c-4182-83a1-f49b799201bc" containerName="payload-processing"
Sep 09 18:51:42.247943 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.247891    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="67634d09-6a9c-4182-83a1-f49b799201bc" containerName="payload-processing"
Sep 09 18:51:42.247943 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.247907    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="af3eb9b8-6844-43b0-be35-f9a64e4a8218" containerName="payload-pre-processing"
Sep 09 18:51:42.247943 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.247919    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b05e05-0b87-4cb1-8732-6771dc3b75d2" containerName="main"
Sep 09 18:51:42.253041 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.253013    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.255910 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.255879    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c5cdc4\"/\"openshift-service-ca.crt\""
Sep 09 18:51:42.255910 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.255899    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c5cdc4\"/\"default-dockercfg-4zs7c\""
Sep 09 18:51:42.256779 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.256757    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c5cdc4\"/\"noten-model-c5cdc4-kserve-self-signed-certs\""
Sep 09 18:51:42.256779 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.256774    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c5cdc4\"/\"kube-root-ca.crt\""
Sep 09 18:51:42.263699 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.263671    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"]
Sep 09 18:51:42.354976 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.354933    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-home\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.354976 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.354983    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.355210 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.355008    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-dshm\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.355210 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.355064    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzbw5\" (UniqueName: \"kubernetes.io/projected/ceb83ad8-35b4-4daa-933e-52b648371cdb-kube-api-access-lzbw5\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.355210 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.355087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-tmp-dir\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.355210 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.355114    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-model-cache\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.456596 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.456555    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-home\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.456767 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.456611    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.456767 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.456645    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-dshm\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.456767 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.456713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzbw5\" (UniqueName: \"kubernetes.io/projected/ceb83ad8-35b4-4daa-933e-52b648371cdb-kube-api-access-lzbw5\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.456767 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.456755    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-tmp-dir\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.456994 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.456794    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-model-cache\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.457110 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.457087    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-tmp-dir\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.457231 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.457209    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-model-cache\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.457294 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.457258    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-home\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.458997 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.458972    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-dshm\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.459187 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.459170    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.465302 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.465259    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzbw5\" (UniqueName: \"kubernetes.io/projected/ceb83ad8-35b4-4daa-933e-52b648371cdb-kube-api-access-lzbw5\") pod \"noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") " pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.564141 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.564032    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:42.843096 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.843065    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"]
Sep 09 18:51:42.845426 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:51:42.845399    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podceb83ad8_35b4_4daa_933e_52b648371cdb.slice/crio-dd156ed84e8ed47d0ac03d78427a7109b4b98276b4ff7bce4a82a609a9fc1b7a WatchSource:0}: Error finding container dd156ed84e8ed47d0ac03d78427a7109b4b98276b4ff7bce4a82a609a9fc1b7a: Status 404 returned error can't find the container with id dd156ed84e8ed47d0ac03d78427a7109b4b98276b4ff7bce4a82a609a9fc1b7a
Sep 09 18:51:42.931334 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.931305    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9" event={"ID":"ceb83ad8-35b4-4daa-933e-52b648371cdb","Type":"ContainerStarted","Data":"a0df3077a928ba6e94371f3c7268806c7b5e89b7b236cffb9212f6c8f4f061e9"}
Sep 09 18:51:42.931485 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.931345    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9" event={"ID":"ceb83ad8-35b4-4daa-933e-52b648371cdb","Type":"ContainerStarted","Data":"dd156ed84e8ed47d0ac03d78427a7109b4b98276b4ff7bce4a82a609a9fc1b7a"}
Sep 09 18:51:42.951602 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:42.951531    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9" podStartSLOduration=0.951493314 podStartE2EDuration="951.493314ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:51:42.951114514 +0000 UTC m=+1191.188484327" watchObservedRunningTime="2026-09-09 18:51:42.951493314 +0000 UTC m=+1191.188863130"
Sep 09 18:51:52.296688 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:52.296651    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 18:51:52.299240 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:52.299214    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 18:51:52.564563 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:52.564435    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:52.564563 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:52.564539    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:52.578305 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:52.578269    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:52.993108 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:52.993063    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:51:53.248449 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:53.248360    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"]
Sep 09 18:51:54.286771 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:51:54.286726    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"]
Sep 09 18:51:55.071735 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:51:55.071699    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-self-signed-certs: secret "noten-model-c5cdc4-kserve-self-signed-certs" not found
Sep 09 18:51:55.071928 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:51:55.071776    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs podName:ceb83ad8-35b4-4daa-933e-52b648371cdb nodeName:}" failed. No retries permitted until 2026-09-09 18:51:55.571760119 +0000 UTC m=+1203.809129915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs") pod "noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9" (UID: "ceb83ad8-35b4-4daa-933e-52b648371cdb") : secret "noten-model-c5cdc4-kserve-self-signed-certs" not found
Sep 09 18:51:55.577961 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:51:55.577922    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-self-signed-certs: secret "noten-model-c5cdc4-kserve-self-signed-certs" not found
Sep 09 18:51:55.578378 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:51:55.578000    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs podName:ceb83ad8-35b4-4daa-933e-52b648371cdb nodeName:}" failed. No retries permitted until 2026-09-09 18:51:56.577985827 +0000 UTC m=+1204.815355620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs") pod "noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9" (UID: "ceb83ad8-35b4-4daa-933e-52b648371cdb") : secret "noten-model-c5cdc4-kserve-self-signed-certs" not found
Sep 09 18:51:56.595174 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:51:56.595129    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-self-signed-certs: secret "noten-model-c5cdc4-kserve-self-signed-certs" not found
Sep 09 18:51:56.595657 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:51:56.595206    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs podName:ceb83ad8-35b4-4daa-933e-52b648371cdb nodeName:}" failed. No retries permitted until 2026-09-09 18:51:58.595191866 +0000 UTC m=+1206.832561658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs") pod "noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9" (UID: "ceb83ad8-35b4-4daa-933e-52b648371cdb") : secret "noten-model-c5cdc4-kserve-self-signed-certs" not found
Sep 09 18:51:58.614960 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:51:58.614928    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-self-signed-certs: secret "noten-model-c5cdc4-kserve-self-signed-certs" not found
Sep 09 18:51:58.615571 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:51:58.615010    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs podName:ceb83ad8-35b4-4daa-933e-52b648371cdb nodeName:}" failed. No retries permitted until 2026-09-09 18:52:02.614993473 +0000 UTC m=+1210.852363266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs") pod "noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9" (UID: "ceb83ad8-35b4-4daa-933e-52b648371cdb") : secret "noten-model-c5cdc4-kserve-self-signed-certs" not found
Sep 09 18:52:02.653012 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:52:02.652971    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-self-signed-certs: secret "noten-model-c5cdc4-kserve-self-signed-certs" not found
Sep 09 18:52:02.655119 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:52:02.653062    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs podName:ceb83ad8-35b4-4daa-933e-52b648371cdb nodeName:}" failed. No retries permitted until 2026-09-09 18:52:10.653044449 +0000 UTC m=+1218.890414242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs") pod "noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9" (UID: "ceb83ad8-35b4-4daa-933e-52b648371cdb") : secret "noten-model-c5cdc4-kserve-self-signed-certs" not found
Sep 09 18:52:03.995516 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:03.995460    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"]
Sep 09 18:52:03.996018 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:03.995781    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8" podUID="3d49c131-9c66-4c5f-abd1-2afe90aeab15" containerName="payload-processing" containerID="cri-o://f59171de851f3e11bfbd87df26cd227d7a5aa0d829789059e218204a30ba5123" gracePeriod=30
Sep 09 18:52:04.247084 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:04.247012    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"
Sep 09 18:52:04.269749 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:04.269683    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2n7s\" (UniqueName: \"kubernetes.io/projected/3d49c131-9c66-4c5f-abd1-2afe90aeab15-kube-api-access-m2n7s\") pod \"3d49c131-9c66-4c5f-abd1-2afe90aeab15\" (UID: \"3d49c131-9c66-4c5f-abd1-2afe90aeab15\") "
Sep 09 18:52:04.270116 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:04.269775    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d49c131-9c66-4c5f-abd1-2afe90aeab15-plugins-config-volume\") pod \"3d49c131-9c66-4c5f-abd1-2afe90aeab15\" (UID: \"3d49c131-9c66-4c5f-abd1-2afe90aeab15\") "
Sep 09 18:52:04.270558 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:04.270527    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d49c131-9c66-4c5f-abd1-2afe90aeab15-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d49c131-9c66-4c5f-abd1-2afe90aeab15" (UID: "3d49c131-9c66-4c5f-abd1-2afe90aeab15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:52:04.272352 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:04.272328    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d49c131-9c66-4c5f-abd1-2afe90aeab15-kube-api-access-m2n7s" (OuterVolumeSpecName: "kube-api-access-m2n7s") pod "3d49c131-9c66-4c5f-abd1-2afe90aeab15" (UID: "3d49c131-9c66-4c5f-abd1-2afe90aeab15"). InnerVolumeSpecName "kube-api-access-m2n7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:52:04.371076 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:04.371042    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2n7s\" (UniqueName: \"kubernetes.io/projected/3d49c131-9c66-4c5f-abd1-2afe90aeab15-kube-api-access-m2n7s\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:04.371076 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:04.371072    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d49c131-9c66-4c5f-abd1-2afe90aeab15-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:05.053053 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:05.053017    2575 generic.go:356] "Generic (PLEG): container finished" podID="3d49c131-9c66-4c5f-abd1-2afe90aeab15" containerID="f59171de851f3e11bfbd87df26cd227d7a5aa0d829789059e218204a30ba5123" exitCode=0
Sep 09 18:52:05.053584 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:05.053067    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"
Sep 09 18:52:05.053584 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:05.053098    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8" event={"ID":"3d49c131-9c66-4c5f-abd1-2afe90aeab15","Type":"ContainerDied","Data":"f59171de851f3e11bfbd87df26cd227d7a5aa0d829789059e218204a30ba5123"}
Sep 09 18:52:05.053584 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:05.053151    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8" event={"ID":"3d49c131-9c66-4c5f-abd1-2afe90aeab15","Type":"ContainerDied","Data":"017c657b5f590d30e4f4c61b35730196ff653f0c6785b244af1a161a596d78b4"}
Sep 09 18:52:05.053584 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:05.053173    2575 scope.go:117] "RemoveContainer" containerID="f59171de851f3e11bfbd87df26cd227d7a5aa0d829789059e218204a30ba5123"
Sep 09 18:52:05.062473 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:05.062446    2575 scope.go:117] "RemoveContainer" containerID="f59171de851f3e11bfbd87df26cd227d7a5aa0d829789059e218204a30ba5123"
Sep 09 18:52:05.062769 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:52:05.062751    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f59171de851f3e11bfbd87df26cd227d7a5aa0d829789059e218204a30ba5123\": container with ID starting with f59171de851f3e11bfbd87df26cd227d7a5aa0d829789059e218204a30ba5123 not found: ID does not exist" containerID="f59171de851f3e11bfbd87df26cd227d7a5aa0d829789059e218204a30ba5123"
Sep 09 18:52:05.062821 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:05.062778    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f59171de851f3e11bfbd87df26cd227d7a5aa0d829789059e218204a30ba5123"} err="failed to get container status \"f59171de851f3e11bfbd87df26cd227d7a5aa0d829789059e218204a30ba5123\": rpc error: code = NotFound desc = could not find container \"f59171de851f3e11bfbd87df26cd227d7a5aa0d829789059e218204a30ba5123\": container with ID starting with f59171de851f3e11bfbd87df26cd227d7a5aa0d829789059e218204a30ba5123 not found: ID does not exist"
Sep 09 18:52:05.082265 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:05.082234    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"]
Sep 09 18:52:05.094328 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:05.094296    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-90f815-6ff958fbff-6h2t8"]
Sep 09 18:52:06.322204 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:06.322164    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d49c131-9c66-4c5f-abd1-2afe90aeab15" path="/var/lib/kubelet/pods/3d49c131-9c66-4c5f-abd1-2afe90aeab15/volumes"
Sep 09 18:52:08.269402 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.269360    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p" podUID="b850a27c-5abe-44d9-af36-99752811e17e" containerName="main" containerID="cri-o://1a2d238f0676eeb66e899f4ab4fd6bedf55b6f0bb58c63034193884ec3c64f12" gracePeriod=45
Sep 09 18:52:08.532707 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.532629    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:52:08.613160 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.613119    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-home\") pod \"b850a27c-5abe-44d9-af36-99752811e17e\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") "
Sep 09 18:52:08.613160 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.613169    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b850a27c-5abe-44d9-af36-99752811e17e-tls-certs\") pod \"b850a27c-5abe-44d9-af36-99752811e17e\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") "
Sep 09 18:52:08.613402 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.613196    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wth9q\" (UniqueName: \"kubernetes.io/projected/b850a27c-5abe-44d9-af36-99752811e17e-kube-api-access-wth9q\") pod \"b850a27c-5abe-44d9-af36-99752811e17e\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") "
Sep 09 18:52:08.613402 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.613215    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-model-cache\") pod \"b850a27c-5abe-44d9-af36-99752811e17e\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") "
Sep 09 18:52:08.613402 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.613265    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-dshm\") pod \"b850a27c-5abe-44d9-af36-99752811e17e\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") "
Sep 09 18:52:08.613402 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.613286    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-tmp-dir\") pod \"b850a27c-5abe-44d9-af36-99752811e17e\" (UID: \"b850a27c-5abe-44d9-af36-99752811e17e\") "
Sep 09 18:52:08.613653 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.613555    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-home" (OuterVolumeSpecName: "home") pod "b850a27c-5abe-44d9-af36-99752811e17e" (UID: "b850a27c-5abe-44d9-af36-99752811e17e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:52:08.613712 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.613646    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b850a27c-5abe-44d9-af36-99752811e17e" (UID: "b850a27c-5abe-44d9-af36-99752811e17e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:52:08.613712 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.613664    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b850a27c-5abe-44d9-af36-99752811e17e" (UID: "b850a27c-5abe-44d9-af36-99752811e17e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:52:08.615660 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.615631    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b850a27c-5abe-44d9-af36-99752811e17e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b850a27c-5abe-44d9-af36-99752811e17e" (UID: "b850a27c-5abe-44d9-af36-99752811e17e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:52:08.615660 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.615631    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b850a27c-5abe-44d9-af36-99752811e17e-kube-api-access-wth9q" (OuterVolumeSpecName: "kube-api-access-wth9q") pod "b850a27c-5abe-44d9-af36-99752811e17e" (UID: "b850a27c-5abe-44d9-af36-99752811e17e"). InnerVolumeSpecName "kube-api-access-wth9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:52:08.615809 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.615704    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-dshm" (OuterVolumeSpecName: "dshm") pod "b850a27c-5abe-44d9-af36-99752811e17e" (UID: "b850a27c-5abe-44d9-af36-99752811e17e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:52:08.714673 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.714638    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:08.714673 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.714666    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b850a27c-5abe-44d9-af36-99752811e17e-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:08.714673 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.714678    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wth9q\" (UniqueName: \"kubernetes.io/projected/b850a27c-5abe-44d9-af36-99752811e17e-kube-api-access-wth9q\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:08.714914 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.714686    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:08.714914 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.714694    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:08.714914 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:08.714703    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b850a27c-5abe-44d9-af36-99752811e17e-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:09.077152 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:09.077117    2575 generic.go:356] "Generic (PLEG): container finished" podID="b850a27c-5abe-44d9-af36-99752811e17e" containerID="1a2d238f0676eeb66e899f4ab4fd6bedf55b6f0bb58c63034193884ec3c64f12" exitCode=0
Sep 09 18:52:09.077356 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:09.077168    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"
Sep 09 18:52:09.077356 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:09.077198    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p" event={"ID":"b850a27c-5abe-44d9-af36-99752811e17e","Type":"ContainerDied","Data":"1a2d238f0676eeb66e899f4ab4fd6bedf55b6f0bb58c63034193884ec3c64f12"}
Sep 09 18:52:09.077356 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:09.077239    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p" event={"ID":"b850a27c-5abe-44d9-af36-99752811e17e","Type":"ContainerDied","Data":"6eef14dc879a5b627d16c57d2cd5ff74e93adf1cb2089e68f0f7fcea816d6280"}
Sep 09 18:52:09.077356 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:09.077256    2575 scope.go:117] "RemoveContainer" containerID="1a2d238f0676eeb66e899f4ab4fd6bedf55b6f0bb58c63034193884ec3c64f12"
Sep 09 18:52:09.086618 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:09.086594    2575 scope.go:117] "RemoveContainer" containerID="1a2d238f0676eeb66e899f4ab4fd6bedf55b6f0bb58c63034193884ec3c64f12"
Sep 09 18:52:09.086964 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:52:09.086939    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a2d238f0676eeb66e899f4ab4fd6bedf55b6f0bb58c63034193884ec3c64f12\": container with ID starting with 1a2d238f0676eeb66e899f4ab4fd6bedf55b6f0bb58c63034193884ec3c64f12 not found: ID does not exist" containerID="1a2d238f0676eeb66e899f4ab4fd6bedf55b6f0bb58c63034193884ec3c64f12"
Sep 09 18:52:09.087032 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:09.086993    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a2d238f0676eeb66e899f4ab4fd6bedf55b6f0bb58c63034193884ec3c64f12"} err="failed to get container status \"1a2d238f0676eeb66e899f4ab4fd6bedf55b6f0bb58c63034193884ec3c64f12\": rpc error: code = NotFound desc = could not find container \"1a2d238f0676eeb66e899f4ab4fd6bedf55b6f0bb58c63034193884ec3c64f12\": container with ID starting with 1a2d238f0676eeb66e899f4ab4fd6bedf55b6f0bb58c63034193884ec3c64f12 not found: ID does not exist"
Sep 09 18:52:09.098759 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:09.098722    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"]
Sep 09 18:52:09.105021 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:09.104995    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-77aadf/test-model-77aadf-kserve-d99d89769-q5t2p"]
Sep 09 18:52:10.321537 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:10.321484    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b850a27c-5abe-44d9-af36-99752811e17e" path="/var/lib/kubelet/pods/b850a27c-5abe-44d9-af36-99752811e17e/volumes"
Sep 09 18:52:10.735239 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:52:10.735201    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-self-signed-certs: secret "noten-model-c5cdc4-kserve-self-signed-certs" not found
Sep 09 18:52:10.735433 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:52:10.735269    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs podName:ceb83ad8-35b4-4daa-933e-52b648371cdb nodeName:}" failed. No retries permitted until 2026-09-09 18:52:26.735254976 +0000 UTC m=+1234.972624769 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs") pod "noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9" (UID: "ceb83ad8-35b4-4daa-933e-52b648371cdb") : secret "noten-model-c5cdc4-kserve-self-signed-certs" not found
Sep 09 18:52:11.025251 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.025112    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9" podUID="ceb83ad8-35b4-4daa-933e-52b648371cdb" containerName="main" containerID="cri-o://a0df3077a928ba6e94371f3c7268806c7b5e89b7b236cffb9212f6c8f4f061e9" gracePeriod=45
Sep 09 18:52:11.294258 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.294229    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:52:11.340294 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.340253    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs\") pod \"ceb83ad8-35b4-4daa-933e-52b648371cdb\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") "
Sep 09 18:52:11.340810 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.340359    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-model-cache\") pod \"ceb83ad8-35b4-4daa-933e-52b648371cdb\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") "
Sep 09 18:52:11.340810 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.340386    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-dshm\") pod \"ceb83ad8-35b4-4daa-933e-52b648371cdb\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") "
Sep 09 18:52:11.340810 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.340413    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzbw5\" (UniqueName: \"kubernetes.io/projected/ceb83ad8-35b4-4daa-933e-52b648371cdb-kube-api-access-lzbw5\") pod \"ceb83ad8-35b4-4daa-933e-52b648371cdb\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") "
Sep 09 18:52:11.340810 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.340450    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-home\") pod \"ceb83ad8-35b4-4daa-933e-52b648371cdb\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") "
Sep 09 18:52:11.340810 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.340476    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-tmp-dir\") pod \"ceb83ad8-35b4-4daa-933e-52b648371cdb\" (UID: \"ceb83ad8-35b4-4daa-933e-52b648371cdb\") "
Sep 09 18:52:11.340810 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.340707    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-model-cache" (OuterVolumeSpecName: "model-cache") pod "ceb83ad8-35b4-4daa-933e-52b648371cdb" (UID: "ceb83ad8-35b4-4daa-933e-52b648371cdb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:52:11.341108 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.340835    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ceb83ad8-35b4-4daa-933e-52b648371cdb" (UID: "ceb83ad8-35b4-4daa-933e-52b648371cdb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:52:11.341108 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.340854    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-home" (OuterVolumeSpecName: "home") pod "ceb83ad8-35b4-4daa-933e-52b648371cdb" (UID: "ceb83ad8-35b4-4daa-933e-52b648371cdb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:52:11.341108 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.340922    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:11.341108 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.340939    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:11.341108 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.340951    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:11.342758 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.342735    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-dshm" (OuterVolumeSpecName: "dshm") pod "ceb83ad8-35b4-4daa-933e-52b648371cdb" (UID: "ceb83ad8-35b4-4daa-933e-52b648371cdb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:52:11.342945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.342925    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb83ad8-35b4-4daa-933e-52b648371cdb-kube-api-access-lzbw5" (OuterVolumeSpecName: "kube-api-access-lzbw5") pod "ceb83ad8-35b4-4daa-933e-52b648371cdb" (UID: "ceb83ad8-35b4-4daa-933e-52b648371cdb"). InnerVolumeSpecName "kube-api-access-lzbw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:52:11.343023 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.343010    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ceb83ad8-35b4-4daa-933e-52b648371cdb" (UID: "ceb83ad8-35b4-4daa-933e-52b648371cdb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:52:11.442466 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.442428    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceb83ad8-35b4-4daa-933e-52b648371cdb-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:11.442466 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.442462    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceb83ad8-35b4-4daa-933e-52b648371cdb-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:11.442466 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:11.442472    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzbw5\" (UniqueName: \"kubernetes.io/projected/ceb83ad8-35b4-4daa-933e-52b648371cdb-kube-api-access-lzbw5\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:12.096320 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:12.096277    2575 generic.go:356] "Generic (PLEG): container finished" podID="ceb83ad8-35b4-4daa-933e-52b648371cdb" containerID="a0df3077a928ba6e94371f3c7268806c7b5e89b7b236cffb9212f6c8f4f061e9" exitCode=0
Sep 09 18:52:12.096320 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:12.096324    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9" event={"ID":"ceb83ad8-35b4-4daa-933e-52b648371cdb","Type":"ContainerDied","Data":"a0df3077a928ba6e94371f3c7268806c7b5e89b7b236cffb9212f6c8f4f061e9"}
Sep 09 18:52:12.096586 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:12.096331    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"
Sep 09 18:52:12.096586 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:12.096357    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9" event={"ID":"ceb83ad8-35b4-4daa-933e-52b648371cdb","Type":"ContainerDied","Data":"dd156ed84e8ed47d0ac03d78427a7109b4b98276b4ff7bce4a82a609a9fc1b7a"}
Sep 09 18:52:12.096586 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:12.096374    2575 scope.go:117] "RemoveContainer" containerID="a0df3077a928ba6e94371f3c7268806c7b5e89b7b236cffb9212f6c8f4f061e9"
Sep 09 18:52:12.106735 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:12.106713    2575 scope.go:117] "RemoveContainer" containerID="a0df3077a928ba6e94371f3c7268806c7b5e89b7b236cffb9212f6c8f4f061e9"
Sep 09 18:52:12.107079 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:52:12.107059    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0df3077a928ba6e94371f3c7268806c7b5e89b7b236cffb9212f6c8f4f061e9\": container with ID starting with a0df3077a928ba6e94371f3c7268806c7b5e89b7b236cffb9212f6c8f4f061e9 not found: ID does not exist" containerID="a0df3077a928ba6e94371f3c7268806c7b5e89b7b236cffb9212f6c8f4f061e9"
Sep 09 18:52:12.107137 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:12.107087    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0df3077a928ba6e94371f3c7268806c7b5e89b7b236cffb9212f6c8f4f061e9"} err="failed to get container status \"a0df3077a928ba6e94371f3c7268806c7b5e89b7b236cffb9212f6c8f4f061e9\": rpc error: code = NotFound desc = could not find container \"a0df3077a928ba6e94371f3c7268806c7b5e89b7b236cffb9212f6c8f4f061e9\": container with ID starting with a0df3077a928ba6e94371f3c7268806c7b5e89b7b236cffb9212f6c8f4f061e9 not found: ID does not exist"
Sep 09 18:52:12.118871 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:12.118840    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"]
Sep 09 18:52:12.124358 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:12.124330    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c5cdc4/noten-model-c5cdc4-kserve-6c5c5b494c-tfwj9"]
Sep 09 18:52:12.321382 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:12.321342    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceb83ad8-35b4-4daa-933e-52b648371cdb" path="/var/lib/kubelet/pods/ceb83ad8-35b4-4daa-933e-52b648371cdb/volumes"
Sep 09 18:52:22.072566 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.072492    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"]
Sep 09 18:52:22.074976 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.074930    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd" podUID="627dada0-b2d6-4669-b7aa-183b47ea2539" containerName="payload-processing" containerID="cri-o://dc35da29a025ebe575b4ff686a30808a25572b77447128ec2c0b726c914131e6" gracePeriod=30
Sep 09 18:52:22.076688 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.076655    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"]
Sep 09 18:52:22.077088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.077058    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh" podUID="b4317af6-a191-4e52-b714-867c12233c03" containerName="payload-pre-processing" containerID="cri-o://6d5e54bb2778fb6a1c33da45a39190ca1555e468dd71c0f9d0e2460e5423f928" gracePeriod=30
Sep 09 18:52:22.369785 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.369750    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"
Sep 09 18:52:22.372806 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.372781    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"
Sep 09 18:52:22.456763 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.456722    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/627dada0-b2d6-4669-b7aa-183b47ea2539-plugins-config-volume\") pod \"627dada0-b2d6-4669-b7aa-183b47ea2539\" (UID: \"627dada0-b2d6-4669-b7aa-183b47ea2539\") "
Sep 09 18:52:22.456952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.456790    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-794qs\" (UniqueName: \"kubernetes.io/projected/627dada0-b2d6-4669-b7aa-183b47ea2539-kube-api-access-794qs\") pod \"627dada0-b2d6-4669-b7aa-183b47ea2539\" (UID: \"627dada0-b2d6-4669-b7aa-183b47ea2539\") "
Sep 09 18:52:22.456952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.456835    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2d9l\" (UniqueName: \"kubernetes.io/projected/b4317af6-a191-4e52-b714-867c12233c03-kube-api-access-x2d9l\") pod \"b4317af6-a191-4e52-b714-867c12233c03\" (UID: \"b4317af6-a191-4e52-b714-867c12233c03\") "
Sep 09 18:52:22.456952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.456874    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4317af6-a191-4e52-b714-867c12233c03-plugins-config-volume\") pod \"b4317af6-a191-4e52-b714-867c12233c03\" (UID: \"b4317af6-a191-4e52-b714-867c12233c03\") "
Sep 09 18:52:22.457215 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.457189    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/627dada0-b2d6-4669-b7aa-183b47ea2539-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "627dada0-b2d6-4669-b7aa-183b47ea2539" (UID: "627dada0-b2d6-4669-b7aa-183b47ea2539"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:52:22.457329 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.457290    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/627dada0-b2d6-4669-b7aa-183b47ea2539-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:22.457393 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.457322    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4317af6-a191-4e52-b714-867c12233c03-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4317af6-a191-4e52-b714-867c12233c03" (UID: "b4317af6-a191-4e52-b714-867c12233c03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:52:22.458964 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.458939    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627dada0-b2d6-4669-b7aa-183b47ea2539-kube-api-access-794qs" (OuterVolumeSpecName: "kube-api-access-794qs") pod "627dada0-b2d6-4669-b7aa-183b47ea2539" (UID: "627dada0-b2d6-4669-b7aa-183b47ea2539"). InnerVolumeSpecName "kube-api-access-794qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:52:22.459072 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.459048    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4317af6-a191-4e52-b714-867c12233c03-kube-api-access-x2d9l" (OuterVolumeSpecName: "kube-api-access-x2d9l") pod "b4317af6-a191-4e52-b714-867c12233c03" (UID: "b4317af6-a191-4e52-b714-867c12233c03"). InnerVolumeSpecName "kube-api-access-x2d9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:52:22.558447 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.558403    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4317af6-a191-4e52-b714-867c12233c03-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:22.558447 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.558440    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-794qs\" (UniqueName: \"kubernetes.io/projected/627dada0-b2d6-4669-b7aa-183b47ea2539-kube-api-access-794qs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:22.558447 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:22.558450    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2d9l\" (UniqueName: \"kubernetes.io/projected/b4317af6-a191-4e52-b714-867c12233c03-kube-api-access-x2d9l\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:52:23.156177 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.156141    2575 generic.go:356] "Generic (PLEG): container finished" podID="b4317af6-a191-4e52-b714-867c12233c03" containerID="6d5e54bb2778fb6a1c33da45a39190ca1555e468dd71c0f9d0e2460e5423f928" exitCode=0
Sep 09 18:52:23.156680 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.156184    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"
Sep 09 18:52:23.156680 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.156227    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh" event={"ID":"b4317af6-a191-4e52-b714-867c12233c03","Type":"ContainerDied","Data":"6d5e54bb2778fb6a1c33da45a39190ca1555e468dd71c0f9d0e2460e5423f928"}
Sep 09 18:52:23.156680 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.156278    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh" event={"ID":"b4317af6-a191-4e52-b714-867c12233c03","Type":"ContainerDied","Data":"8ca8f6ae850b8e558a284b73156dd5202a3f250207dea38a9cafb103a183ea47"}
Sep 09 18:52:23.156680 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.156307    2575 scope.go:117] "RemoveContainer" containerID="6d5e54bb2778fb6a1c33da45a39190ca1555e468dd71c0f9d0e2460e5423f928"
Sep 09 18:52:23.157865 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.157838    2575 generic.go:356] "Generic (PLEG): container finished" podID="627dada0-b2d6-4669-b7aa-183b47ea2539" containerID="dc35da29a025ebe575b4ff686a30808a25572b77447128ec2c0b726c914131e6" exitCode=0
Sep 09 18:52:23.157938 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.157888    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"
Sep 09 18:52:23.157938 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.157920    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd" event={"ID":"627dada0-b2d6-4669-b7aa-183b47ea2539","Type":"ContainerDied","Data":"dc35da29a025ebe575b4ff686a30808a25572b77447128ec2c0b726c914131e6"}
Sep 09 18:52:23.158044 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.157990    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd" event={"ID":"627dada0-b2d6-4669-b7aa-183b47ea2539","Type":"ContainerDied","Data":"86f16b29fdd65195fae81608c24435244aa6f7b281ce113480813f3a1083c932"}
Sep 09 18:52:23.166082 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.166053    2575 scope.go:117] "RemoveContainer" containerID="6d5e54bb2778fb6a1c33da45a39190ca1555e468dd71c0f9d0e2460e5423f928"
Sep 09 18:52:23.166435 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:52:23.166401    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5e54bb2778fb6a1c33da45a39190ca1555e468dd71c0f9d0e2460e5423f928\": container with ID starting with 6d5e54bb2778fb6a1c33da45a39190ca1555e468dd71c0f9d0e2460e5423f928 not found: ID does not exist" containerID="6d5e54bb2778fb6a1c33da45a39190ca1555e468dd71c0f9d0e2460e5423f928"
Sep 09 18:52:23.166557 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.166442    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5e54bb2778fb6a1c33da45a39190ca1555e468dd71c0f9d0e2460e5423f928"} err="failed to get container status \"6d5e54bb2778fb6a1c33da45a39190ca1555e468dd71c0f9d0e2460e5423f928\": rpc error: code = NotFound desc = could not find container \"6d5e54bb2778fb6a1c33da45a39190ca1555e468dd71c0f9d0e2460e5423f928\": container with ID starting with 6d5e54bb2778fb6a1c33da45a39190ca1555e468dd71c0f9d0e2460e5423f928 not found: ID does not exist"
Sep 09 18:52:23.166557 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.166468    2575 scope.go:117] "RemoveContainer" containerID="dc35da29a025ebe575b4ff686a30808a25572b77447128ec2c0b726c914131e6"
Sep 09 18:52:23.175291 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.175271    2575 scope.go:117] "RemoveContainer" containerID="dc35da29a025ebe575b4ff686a30808a25572b77447128ec2c0b726c914131e6"
Sep 09 18:52:23.175586 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:52:23.175565    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc35da29a025ebe575b4ff686a30808a25572b77447128ec2c0b726c914131e6\": container with ID starting with dc35da29a025ebe575b4ff686a30808a25572b77447128ec2c0b726c914131e6 not found: ID does not exist" containerID="dc35da29a025ebe575b4ff686a30808a25572b77447128ec2c0b726c914131e6"
Sep 09 18:52:23.175645 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.175598    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc35da29a025ebe575b4ff686a30808a25572b77447128ec2c0b726c914131e6"} err="failed to get container status \"dc35da29a025ebe575b4ff686a30808a25572b77447128ec2c0b726c914131e6\": rpc error: code = NotFound desc = could not find container \"dc35da29a025ebe575b4ff686a30808a25572b77447128ec2c0b726c914131e6\": container with ID starting with dc35da29a025ebe575b4ff686a30808a25572b77447128ec2c0b726c914131e6 not found: ID does not exist"
Sep 09 18:52:23.183528 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.183480    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"]
Sep 09 18:52:23.189259 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.189227    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-77aadf-85bff8c598-vr6gh"]
Sep 09 18:52:23.199630 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.199598    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"]
Sep 09 18:52:23.203156 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:23.203127    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-77aadf-c9dccc6ff-qxrsd"]
Sep 09 18:52:24.320627 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:24.320593    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="627dada0-b2d6-4669-b7aa-183b47ea2539" path="/var/lib/kubelet/pods/627dada0-b2d6-4669-b7aa-183b47ea2539/volumes"
Sep 09 18:52:24.321013 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:24.320975    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4317af6-a191-4e52-b714-867c12233c03" path="/var/lib/kubelet/pods/b4317af6-a191-4e52-b714-867c12233c03/volumes"
Sep 09 18:52:35.060305 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060260    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"]
Sep 09 18:52:35.060802 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060694    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d49c131-9c66-4c5f-abd1-2afe90aeab15" containerName="payload-processing"
Sep 09 18:52:35.060802 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060706    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d49c131-9c66-4c5f-abd1-2afe90aeab15" containerName="payload-processing"
Sep 09 18:52:35.060802 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060724    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4317af6-a191-4e52-b714-867c12233c03" containerName="payload-pre-processing"
Sep 09 18:52:35.060802 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060729    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4317af6-a191-4e52-b714-867c12233c03" containerName="payload-pre-processing"
Sep 09 18:52:35.060802 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060737    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="627dada0-b2d6-4669-b7aa-183b47ea2539" containerName="payload-processing"
Sep 09 18:52:35.060802 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060743    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="627dada0-b2d6-4669-b7aa-183b47ea2539" containerName="payload-processing"
Sep 09 18:52:35.060802 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060758    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b850a27c-5abe-44d9-af36-99752811e17e" containerName="main"
Sep 09 18:52:35.060802 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060763    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b850a27c-5abe-44d9-af36-99752811e17e" containerName="main"
Sep 09 18:52:35.060802 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060772    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceb83ad8-35b4-4daa-933e-52b648371cdb" containerName="main"
Sep 09 18:52:35.060802 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060777    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceb83ad8-35b4-4daa-933e-52b648371cdb" containerName="main"
Sep 09 18:52:35.061111 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060844    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="627dada0-b2d6-4669-b7aa-183b47ea2539" containerName="payload-processing"
Sep 09 18:52:35.061111 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060851    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4317af6-a191-4e52-b714-867c12233c03" containerName="payload-pre-processing"
Sep 09 18:52:35.061111 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060860    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b850a27c-5abe-44d9-af36-99752811e17e" containerName="main"
Sep 09 18:52:35.061111 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060865    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d49c131-9c66-4c5f-abd1-2afe90aeab15" containerName="payload-processing"
Sep 09 18:52:35.061111 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.060870    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceb83ad8-35b4-4daa-933e-52b648371cdb" containerName="main"
Sep 09 18:52:35.064244 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.064194    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"
Sep 09 18:52:35.066885 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.066851    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8820e118-7a7c-4102-a4cd-b0f52f52d4b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm\" (UID: \"8820e118-7a7c-4102-a4cd-b0f52f52d4b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"
Sep 09 18:52:35.066885 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.066877    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-282de3-dockercfg-bqwjc\""
Sep 09 18:52:35.067101 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.066942    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46nm7\" (UniqueName: \"kubernetes.io/projected/8820e118-7a7c-4102-a4cd-b0f52f52d4b4-kube-api-access-46nm7\") pod \"payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm\" (UID: \"8820e118-7a7c-4102-a4cd-b0f52f52d4b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"
Sep 09 18:52:35.067101 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.066956    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-282de3\""
Sep 09 18:52:35.074683 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.074649    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"]
Sep 09 18:52:35.078376 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.078352    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"]
Sep 09 18:52:35.078538 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.078445    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"
Sep 09 18:52:35.093431 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.093397    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"]
Sep 09 18:52:35.168061 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.168020    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8820e118-7a7c-4102-a4cd-b0f52f52d4b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm\" (UID: \"8820e118-7a7c-4102-a4cd-b0f52f52d4b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"
Sep 09 18:52:35.168268 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.168245    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce8222c-e784-4418-be86-5ba581e970b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn\" (UID: \"bce8222c-e784-4418-be86-5ba581e970b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"
Sep 09 18:52:35.168341 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.168291    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46nm7\" (UniqueName: \"kubernetes.io/projected/8820e118-7a7c-4102-a4cd-b0f52f52d4b4-kube-api-access-46nm7\") pod \"payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm\" (UID: \"8820e118-7a7c-4102-a4cd-b0f52f52d4b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"
Sep 09 18:52:35.168423 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.168397    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k5hd\" (UniqueName: \"kubernetes.io/projected/bce8222c-e784-4418-be86-5ba581e970b7-kube-api-access-6k5hd\") pod \"payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn\" (UID: \"bce8222c-e784-4418-be86-5ba581e970b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"
Sep 09 18:52:35.168787 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.168771    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8820e118-7a7c-4102-a4cd-b0f52f52d4b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm\" (UID: \"8820e118-7a7c-4102-a4cd-b0f52f52d4b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"
Sep 09 18:52:35.177266 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.177229    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46nm7\" (UniqueName: \"kubernetes.io/projected/8820e118-7a7c-4102-a4cd-b0f52f52d4b4-kube-api-access-46nm7\") pod \"payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm\" (UID: \"8820e118-7a7c-4102-a4cd-b0f52f52d4b4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"
Sep 09 18:52:35.268900 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.268856    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce8222c-e784-4418-be86-5ba581e970b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn\" (UID: \"bce8222c-e784-4418-be86-5ba581e970b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"
Sep 09 18:52:35.269111 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.269023    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k5hd\" (UniqueName: \"kubernetes.io/projected/bce8222c-e784-4418-be86-5ba581e970b7-kube-api-access-6k5hd\") pod \"payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn\" (UID: \"bce8222c-e784-4418-be86-5ba581e970b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"
Sep 09 18:52:35.269682 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.269657    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce8222c-e784-4418-be86-5ba581e970b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn\" (UID: \"bce8222c-e784-4418-be86-5ba581e970b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"
Sep 09 18:52:35.277088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.277063    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k5hd\" (UniqueName: \"kubernetes.io/projected/bce8222c-e784-4418-be86-5ba581e970b7-kube-api-access-6k5hd\") pod \"payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn\" (UID: \"bce8222c-e784-4418-be86-5ba581e970b7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"
Sep 09 18:52:35.375889 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.375843    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"
Sep 09 18:52:35.388274 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.388240    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"
Sep 09 18:52:35.652879 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.652845    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"]
Sep 09 18:52:35.654754 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:52:35.654718    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8820e118_7a7c_4102_a4cd_b0f52f52d4b4.slice/crio-dde28c3353dee7633d442fe0d510c70e064ecc09792d0951ba6dbf11a0709f64 WatchSource:0}: Error finding container dde28c3353dee7633d442fe0d510c70e064ecc09792d0951ba6dbf11a0709f64: Status 404 returned error can't find the container with id dde28c3353dee7633d442fe0d510c70e064ecc09792d0951ba6dbf11a0709f64
Sep 09 18:52:35.682943 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:35.682918    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"]
Sep 09 18:52:35.686765 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:52:35.686735    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce8222c_e784_4418_be86_5ba581e970b7.slice/crio-bc068ca137e96014aaec9cc9afe30ae8ceabfe785c79a1356ed6c76eb6967fd6 WatchSource:0}: Error finding container bc068ca137e96014aaec9cc9afe30ae8ceabfe785c79a1356ed6c76eb6967fd6: Status 404 returned error can't find the container with id bc068ca137e96014aaec9cc9afe30ae8ceabfe785c79a1356ed6c76eb6967fd6
Sep 09 18:52:36.227466 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:36.227413    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm" event={"ID":"8820e118-7a7c-4102-a4cd-b0f52f52d4b4","Type":"ContainerStarted","Data":"dc49265753f1c1d63b6e0e18e780836302cc6796e0ac20e8a11117ba5838cb3f"}
Sep 09 18:52:36.228012 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:36.227485    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm" event={"ID":"8820e118-7a7c-4102-a4cd-b0f52f52d4b4","Type":"ContainerStarted","Data":"dde28c3353dee7633d442fe0d510c70e064ecc09792d0951ba6dbf11a0709f64"}
Sep 09 18:52:36.228012 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:36.227627    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"
Sep 09 18:52:36.229338 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:36.229301    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn" event={"ID":"bce8222c-e784-4418-be86-5ba581e970b7","Type":"ContainerStarted","Data":"25869d1a1feb63e7b7f3381fa53860afa2f4bf5f4e25f39d8f663377a37105ab"}
Sep 09 18:52:36.229464 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:36.229354    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn" event={"ID":"bce8222c-e784-4418-be86-5ba581e970b7","Type":"ContainerStarted","Data":"bc068ca137e96014aaec9cc9afe30ae8ceabfe785c79a1356ed6c76eb6967fd6"}
Sep 09 18:52:36.229464 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:36.229377    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"
Sep 09 18:52:36.254641 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:36.254576    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm" podStartSLOduration=1.254554811 podStartE2EDuration="1.254554811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:52:36.252958186 +0000 UTC m=+1244.490328003" watchObservedRunningTime="2026-09-09 18:52:36.254554811 +0000 UTC m=+1244.491924627"
Sep 09 18:52:36.281880 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:36.281827    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn" podStartSLOduration=1.281808336 podStartE2EDuration="1.281808336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:52:36.28063579 +0000 UTC m=+1244.518005611" watchObservedRunningTime="2026-09-09 18:52:36.281808336 +0000 UTC m=+1244.519178151"
Sep 09 18:52:38.248142 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:38.248105    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"]
Sep 09 18:52:38.251992 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:38.251957    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"
Sep 09 18:52:38.256686 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:38.256651    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ddaae1a6-dockercfg-dvjkg\""
Sep 09 18:52:38.256686 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:38.256661    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ddaae1a6\""
Sep 09 18:52:38.268858 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:38.268824    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"]
Sep 09 18:52:38.303009 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:38.302972    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvcbx\" (UniqueName: \"kubernetes.io/projected/9d5ff109-e983-4be6-9c15-ac24736dbb2a-kube-api-access-fvcbx\") pod \"payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265\" (UID: \"9d5ff109-e983-4be6-9c15-ac24736dbb2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"
Sep 09 18:52:38.303194 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:38.303101    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5ff109-e983-4be6-9c15-ac24736dbb2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265\" (UID: \"9d5ff109-e983-4be6-9c15-ac24736dbb2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"
Sep 09 18:52:38.404057 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:38.404016    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvcbx\" (UniqueName: \"kubernetes.io/projected/9d5ff109-e983-4be6-9c15-ac24736dbb2a-kube-api-access-fvcbx\") pod \"payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265\" (UID: \"9d5ff109-e983-4be6-9c15-ac24736dbb2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"
Sep 09 18:52:38.404246 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:38.404107    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5ff109-e983-4be6-9c15-ac24736dbb2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265\" (UID: \"9d5ff109-e983-4be6-9c15-ac24736dbb2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"
Sep 09 18:52:38.404902 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:38.404881    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5ff109-e983-4be6-9c15-ac24736dbb2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265\" (UID: \"9d5ff109-e983-4be6-9c15-ac24736dbb2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"
Sep 09 18:52:38.414518 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:38.414465    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvcbx\" (UniqueName: \"kubernetes.io/projected/9d5ff109-e983-4be6-9c15-ac24736dbb2a-kube-api-access-fvcbx\") pod \"payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265\" (UID: \"9d5ff109-e983-4be6-9c15-ac24736dbb2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"
Sep 09 18:52:38.564590 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:38.564452    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"
Sep 09 18:52:38.838985 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:38.838960    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"]
Sep 09 18:52:38.840979 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:52:38.840952    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5ff109_e983_4be6_9c15_ac24736dbb2a.slice/crio-e0d867e8d573ba19acbc80e2d5e94c6f7442b10433996cbf8e5a9c03e54488c9 WatchSource:0}: Error finding container e0d867e8d573ba19acbc80e2d5e94c6f7442b10433996cbf8e5a9c03e54488c9: Status 404 returned error can't find the container with id e0d867e8d573ba19acbc80e2d5e94c6f7442b10433996cbf8e5a9c03e54488c9
Sep 09 18:52:39.249806 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:39.249770    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265" event={"ID":"9d5ff109-e983-4be6-9c15-ac24736dbb2a","Type":"ContainerStarted","Data":"fc40ce2e9dcc5ea45115275ba649b7b6c3c2dd756036c8010a5a72d4103bc207"}
Sep 09 18:52:39.250213 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:39.249826    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265" event={"ID":"9d5ff109-e983-4be6-9c15-ac24736dbb2a","Type":"ContainerStarted","Data":"e0d867e8d573ba19acbc80e2d5e94c6f7442b10433996cbf8e5a9c03e54488c9"}
Sep 09 18:52:39.250213 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:39.249857    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"
Sep 09 18:52:39.269299 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:39.269235    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265" podStartSLOduration=1.269215848 podStartE2EDuration="1.269215848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:52:39.266917036 +0000 UTC m=+1247.504286852" watchObservedRunningTime="2026-09-09 18:52:39.269215848 +0000 UTC m=+1247.506585709"
Sep 09 18:52:47.237465 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:47.237381    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"
Sep 09 18:52:47.237465 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:47.237465    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"
Sep 09 18:52:50.257697 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:52:50.257661    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"
Sep 09 18:53:04.240115 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.240071    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"]
Sep 09 18:53:04.245442 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.245412    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"
Sep 09 18:53:04.248476 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.248444    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-0e012d-dockercfg-df7tz\""
Sep 09 18:53:04.248659 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.248530    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-0e012d\""
Sep 09 18:53:04.261234 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.261199    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"]
Sep 09 18:53:04.273706 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.273665    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"]
Sep 09 18:53:04.277422 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.277399    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"
Sep 09 18:53:04.287743 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.287702    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"]
Sep 09 18:53:04.355195 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.355150    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a88570-392e-4cd0-9528-c541202f5a70-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9\" (UID: \"15a88570-392e-4cd0-9528-c541202f5a70\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"
Sep 09 18:53:04.355195 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.355200    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llq4r\" (UniqueName: \"kubernetes.io/projected/15a88570-392e-4cd0-9528-c541202f5a70-kube-api-access-llq4r\") pod \"payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9\" (UID: \"15a88570-392e-4cd0-9528-c541202f5a70\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"
Sep 09 18:53:04.456768 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.456728    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c47ad7-8b27-4e6b-82fd-8c754b922373-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps\" (UID: \"b2c47ad7-8b27-4e6b-82fd-8c754b922373\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"
Sep 09 18:53:04.456941 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.456780    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a88570-392e-4cd0-9528-c541202f5a70-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9\" (UID: \"15a88570-392e-4cd0-9528-c541202f5a70\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"
Sep 09 18:53:04.456941 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.456811    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llq4r\" (UniqueName: \"kubernetes.io/projected/15a88570-392e-4cd0-9528-c541202f5a70-kube-api-access-llq4r\") pod \"payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9\" (UID: \"15a88570-392e-4cd0-9528-c541202f5a70\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"
Sep 09 18:53:04.456941 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.456833    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7dn\" (UniqueName: \"kubernetes.io/projected/b2c47ad7-8b27-4e6b-82fd-8c754b922373-kube-api-access-rc7dn\") pod \"payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps\" (UID: \"b2c47ad7-8b27-4e6b-82fd-8c754b922373\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"
Sep 09 18:53:04.457466 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.457443    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a88570-392e-4cd0-9528-c541202f5a70-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9\" (UID: \"15a88570-392e-4cd0-9528-c541202f5a70\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"
Sep 09 18:53:04.470096 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.470055    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llq4r\" (UniqueName: \"kubernetes.io/projected/15a88570-392e-4cd0-9528-c541202f5a70-kube-api-access-llq4r\") pod \"payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9\" (UID: \"15a88570-392e-4cd0-9528-c541202f5a70\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"
Sep 09 18:53:04.556409 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.556319    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"
Sep 09 18:53:04.557692 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.557658    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c47ad7-8b27-4e6b-82fd-8c754b922373-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps\" (UID: \"b2c47ad7-8b27-4e6b-82fd-8c754b922373\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"
Sep 09 18:53:04.557809 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.557709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7dn\" (UniqueName: \"kubernetes.io/projected/b2c47ad7-8b27-4e6b-82fd-8c754b922373-kube-api-access-rc7dn\") pod \"payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps\" (UID: \"b2c47ad7-8b27-4e6b-82fd-8c754b922373\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"
Sep 09 18:53:04.558330 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.558311    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c47ad7-8b27-4e6b-82fd-8c754b922373-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps\" (UID: \"b2c47ad7-8b27-4e6b-82fd-8c754b922373\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"
Sep 09 18:53:04.567783 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.567759    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7dn\" (UniqueName: \"kubernetes.io/projected/b2c47ad7-8b27-4e6b-82fd-8c754b922373-kube-api-access-rc7dn\") pod \"payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps\" (UID: \"b2c47ad7-8b27-4e6b-82fd-8c754b922373\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"
Sep 09 18:53:04.591551 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.591492    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"
Sep 09 18:53:04.836576 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.836545    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"]
Sep 09 18:53:04.839011 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:53:04.838977    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a88570_392e_4cd0_9528_c541202f5a70.slice/crio-832b55892da4d304f9f0e560962cf8497bd1951d7bbcd37733a72de54180c92e WatchSource:0}: Error finding container 832b55892da4d304f9f0e560962cf8497bd1951d7bbcd37733a72de54180c92e: Status 404 returned error can't find the container with id 832b55892da4d304f9f0e560962cf8497bd1951d7bbcd37733a72de54180c92e
Sep 09 18:53:04.893065 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:04.891034    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"]
Sep 09 18:53:05.395911 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:05.395858    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps" event={"ID":"b2c47ad7-8b27-4e6b-82fd-8c754b922373","Type":"ContainerStarted","Data":"4116f5916dda37323092bdd0cbd45ad9ec7cf3e7018a5a0e58d47428ce4a4d70"}
Sep 09 18:53:05.396418 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:05.395927    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps" event={"ID":"b2c47ad7-8b27-4e6b-82fd-8c754b922373","Type":"ContainerStarted","Data":"754012caddad17dba80ea22ee5d13035b27645a17b417355541a56683b6c847f"}
Sep 09 18:53:05.396418 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:05.396077    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"
Sep 09 18:53:05.397740 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:05.397716    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9" event={"ID":"15a88570-392e-4cd0-9528-c541202f5a70","Type":"ContainerStarted","Data":"c56ca9cc89986d618050968d9337f5fdab49853923473a7568e3e0e03e972557"}
Sep 09 18:53:05.397872 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:05.397761    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9" event={"ID":"15a88570-392e-4cd0-9528-c541202f5a70","Type":"ContainerStarted","Data":"832b55892da4d304f9f0e560962cf8497bd1951d7bbcd37733a72de54180c92e"}
Sep 09 18:53:05.397872 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:05.397802    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"
Sep 09 18:53:05.413809 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:05.413748    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps" podStartSLOduration=1.4137295650000001 podStartE2EDuration="1.413729565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:53:05.411366698 +0000 UTC m=+1273.648736521" watchObservedRunningTime="2026-09-09 18:53:05.413729565 +0000 UTC m=+1273.651099382"
Sep 09 18:53:05.429804 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:05.429748    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9" podStartSLOduration=1.42973115 podStartE2EDuration="1.42973115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:53:05.427698034 +0000 UTC m=+1273.665067850" watchObservedRunningTime="2026-09-09 18:53:05.42973115 +0000 UTC m=+1273.667101011"
Sep 09 18:53:16.404688 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:16.404655    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"
Sep 09 18:53:16.405172 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:16.404981    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"
Sep 09 18:53:19.705031 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:19.704989    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"]
Sep 09 18:53:19.705487 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:19.705326    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265" podUID="9d5ff109-e983-4be6-9c15-ac24736dbb2a" containerName="payload-pre-processing" containerID="cri-o://fc40ce2e9dcc5ea45115275ba649b7b6c3c2dd756036c8010a5a72d4103bc207" gracePeriod=30
Sep 09 18:53:19.984125 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:19.984093    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"
Sep 09 18:53:20.023228 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.023192    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5ff109-e983-4be6-9c15-ac24736dbb2a-plugins-config-volume\") pod \"9d5ff109-e983-4be6-9c15-ac24736dbb2a\" (UID: \"9d5ff109-e983-4be6-9c15-ac24736dbb2a\") "
Sep 09 18:53:20.023429 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.023265    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvcbx\" (UniqueName: \"kubernetes.io/projected/9d5ff109-e983-4be6-9c15-ac24736dbb2a-kube-api-access-fvcbx\") pod \"9d5ff109-e983-4be6-9c15-ac24736dbb2a\" (UID: \"9d5ff109-e983-4be6-9c15-ac24736dbb2a\") "
Sep 09 18:53:20.023654 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.023630    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5ff109-e983-4be6-9c15-ac24736dbb2a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d5ff109-e983-4be6-9c15-ac24736dbb2a" (UID: "9d5ff109-e983-4be6-9c15-ac24736dbb2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:53:20.025540 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.025489    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5ff109-e983-4be6-9c15-ac24736dbb2a-kube-api-access-fvcbx" (OuterVolumeSpecName: "kube-api-access-fvcbx") pod "9d5ff109-e983-4be6-9c15-ac24736dbb2a" (UID: "9d5ff109-e983-4be6-9c15-ac24736dbb2a"). InnerVolumeSpecName "kube-api-access-fvcbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:53:20.124362 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.124319    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d5ff109-e983-4be6-9c15-ac24736dbb2a-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:53:20.124362 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.124354    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvcbx\" (UniqueName: \"kubernetes.io/projected/9d5ff109-e983-4be6-9c15-ac24736dbb2a-kube-api-access-fvcbx\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:53:20.478101 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.478067    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d5ff109-e983-4be6-9c15-ac24736dbb2a" containerID="fc40ce2e9dcc5ea45115275ba649b7b6c3c2dd756036c8010a5a72d4103bc207" exitCode=0
Sep 09 18:53:20.478285 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.478124    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"
Sep 09 18:53:20.478285 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.478151    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265" event={"ID":"9d5ff109-e983-4be6-9c15-ac24736dbb2a","Type":"ContainerDied","Data":"fc40ce2e9dcc5ea45115275ba649b7b6c3c2dd756036c8010a5a72d4103bc207"}
Sep 09 18:53:20.478285 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.478202    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265" event={"ID":"9d5ff109-e983-4be6-9c15-ac24736dbb2a","Type":"ContainerDied","Data":"e0d867e8d573ba19acbc80e2d5e94c6f7442b10433996cbf8e5a9c03e54488c9"}
Sep 09 18:53:20.478285 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.478224    2575 scope.go:117] "RemoveContainer" containerID="fc40ce2e9dcc5ea45115275ba649b7b6c3c2dd756036c8010a5a72d4103bc207"
Sep 09 18:53:20.487360 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.487340    2575 scope.go:117] "RemoveContainer" containerID="fc40ce2e9dcc5ea45115275ba649b7b6c3c2dd756036c8010a5a72d4103bc207"
Sep 09 18:53:20.487726 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:53:20.487701    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc40ce2e9dcc5ea45115275ba649b7b6c3c2dd756036c8010a5a72d4103bc207\": container with ID starting with fc40ce2e9dcc5ea45115275ba649b7b6c3c2dd756036c8010a5a72d4103bc207 not found: ID does not exist" containerID="fc40ce2e9dcc5ea45115275ba649b7b6c3c2dd756036c8010a5a72d4103bc207"
Sep 09 18:53:20.487797 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.487744    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc40ce2e9dcc5ea45115275ba649b7b6c3c2dd756036c8010a5a72d4103bc207"} err="failed to get container status \"fc40ce2e9dcc5ea45115275ba649b7b6c3c2dd756036c8010a5a72d4103bc207\": rpc error: code = NotFound desc = could not find container \"fc40ce2e9dcc5ea45115275ba649b7b6c3c2dd756036c8010a5a72d4103bc207\": container with ID starting with fc40ce2e9dcc5ea45115275ba649b7b6c3c2dd756036c8010a5a72d4103bc207 not found: ID does not exist"
Sep 09 18:53:20.496085 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.496056    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"]
Sep 09 18:53:20.498092 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:20.498071    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ddaae1a6-b584f478b-zq265"]
Sep 09 18:53:22.322166 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:22.322122    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d5ff109-e983-4be6-9c15-ac24736dbb2a" path="/var/lib/kubelet/pods/9d5ff109-e983-4be6-9c15-ac24736dbb2a/volumes"
Sep 09 18:53:29.760211 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.760174    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"]
Sep 09 18:53:29.760799 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.760778    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d5ff109-e983-4be6-9c15-ac24736dbb2a" containerName="payload-pre-processing"
Sep 09 18:53:29.760885 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.760802    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d5ff109-e983-4be6-9c15-ac24736dbb2a" containerName="payload-pre-processing"
Sep 09 18:53:29.760947 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.760938    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d5ff109-e983-4be6-9c15-ac24736dbb2a" containerName="payload-pre-processing"
Sep 09 18:53:29.765436 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.765407    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:29.768382 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.768355    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-282de3\"/\"default-dockercfg-bjp96\""
Sep 09 18:53:29.769491 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.769460    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-282de3\"/\"kube-root-ca.crt\""
Sep 09 18:53:29.769491 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.769473    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-282de3\"/\"openshift-service-ca.crt\""
Sep 09 18:53:29.769690 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.769510    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-282de3\"/\"auth-test-model-282de3-kserve-self-signed-certs\""
Sep 09 18:53:29.777331 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.777299    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"]
Sep 09 18:53:29.919978 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.919923    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x2ms\" (UniqueName: \"kubernetes.io/projected/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-kube-api-access-9x2ms\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:29.919978 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.919980    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-tmp-dir\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:29.920214 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.920105    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-home\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:29.920214 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.920143    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-tls-certs\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:29.920289 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.920229    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-model-cache\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:29.920289 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:29.920261    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-dshm\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:30.021003 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.020886    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-dshm\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:30.021003 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.020947    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x2ms\" (UniqueName: \"kubernetes.io/projected/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-kube-api-access-9x2ms\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:30.021287 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.021066    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-tmp-dir\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:30.021287 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.021161    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-home\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:30.021287 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.021183    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-tls-certs\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:30.021287 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.021264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-model-cache\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:30.021459 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.021420    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-tmp-dir\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:30.021545 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.021495    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-home\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:30.021655 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.021626    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-model-cache\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:30.023308 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.023286    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-dshm\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:30.023642 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.023626    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-tls-certs\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:30.029366 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.029346    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x2ms\" (UniqueName: \"kubernetes.io/projected/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-kube-api-access-9x2ms\") pod \"auth-test-model-282de3-kserve-5987d676bc-8l52l\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") " pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:30.075477 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.075430    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:30.356419 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.356389    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"]
Sep 09 18:53:30.359318 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:53:30.359288    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4095c852_bb8d_4cfa_97e4_d90ae2e385ac.slice/crio-628911873e2a1f6b7a8a646c0205da039516649745a75a119eadd9c1dbc931e2 WatchSource:0}: Error finding container 628911873e2a1f6b7a8a646c0205da039516649745a75a119eadd9c1dbc931e2: Status 404 returned error can't find the container with id 628911873e2a1f6b7a8a646c0205da039516649745a75a119eadd9c1dbc931e2
Sep 09 18:53:30.540574 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.540539    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l" event={"ID":"4095c852-bb8d-4cfa-97e4-d90ae2e385ac","Type":"ContainerStarted","Data":"09c9c86902a2e30ec731e073e3505cb01ac38d1660a48f414f55311fd63bcaf5"}
Sep 09 18:53:30.540755 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.540582    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l" event={"ID":"4095c852-bb8d-4cfa-97e4-d90ae2e385ac","Type":"ContainerStarted","Data":"628911873e2a1f6b7a8a646c0205da039516649745a75a119eadd9c1dbc931e2"}
Sep 09 18:53:30.559340 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:30.559280    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l" podStartSLOduration=1.559262404 podStartE2EDuration="1.559262404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:53:30.557168985 +0000 UTC m=+1298.794538801" watchObservedRunningTime="2026-09-09 18:53:30.559262404 +0000 UTC m=+1298.796632219"
Sep 09 18:53:40.076543 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:40.076480    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:40.077076 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:40.076565    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:40.089698 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:40.089666    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:40.626367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:40.626334    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:53:45.642333 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.642293    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"]
Sep 09 18:53:45.647325 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.647298    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.651092 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.651059    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e012d\"/\"auth-test-model-0e012d-kserve-self-signed-certs\""
Sep 09 18:53:45.651272 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.651129    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e012d\"/\"default-dockercfg-6kjms\""
Sep 09 18:53:45.651272 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.651153    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e012d\"/\"openshift-service-ca.crt\""
Sep 09 18:53:45.651272 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.651177    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e012d\"/\"kube-root-ca.crt\""
Sep 09 18:53:45.657669 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.657638    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"]
Sep 09 18:53:45.692338 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.692291    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbqj8\" (UniqueName: \"kubernetes.io/projected/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-kube-api-access-mbqj8\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.692338 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.692343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-home\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.692653 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.692360    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-tmp-dir\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.692653 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.692444    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-dshm\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.692653 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.692549    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-model-cache\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.692772 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.692660    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-tls-certs\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.794078 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.794038    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-tls-certs\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.794289 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.794087    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbqj8\" (UniqueName: \"kubernetes.io/projected/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-kube-api-access-mbqj8\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.794289 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.794213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-home\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.794289 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.794255    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-tmp-dir\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.794466 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.794299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-dshm\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.794563 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.794467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-model-cache\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.794700 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.794663    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-home\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.794814 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.794677    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-tmp-dir\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.794814 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.794784    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-model-cache\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.796575 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.796555    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-dshm\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.796926 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.796906    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-tls-certs\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.802719 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.802693    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbqj8\" (UniqueName: \"kubernetes.io/projected/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-kube-api-access-mbqj8\") pod \"auth-test-model-0e012d-kserve-864c5968f9-xdt4g\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:45.960708 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:45.960613    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:46.246528 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:46.246479    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"]
Sep 09 18:53:46.248862 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:53:46.248830    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f9e3ab6_23f7_4e1d_986f_491179ef50dd.slice/crio-fa0c3830bd0438e74107834dcd6c6d8790dd9cc149f582be919faaa0b0c4290b WatchSource:0}: Error finding container fa0c3830bd0438e74107834dcd6c6d8790dd9cc149f582be919faaa0b0c4290b: Status 404 returned error can't find the container with id fa0c3830bd0438e74107834dcd6c6d8790dd9cc149f582be919faaa0b0c4290b
Sep 09 18:53:46.648129 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:46.648088    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g" event={"ID":"3f9e3ab6-23f7-4e1d-986f-491179ef50dd","Type":"ContainerStarted","Data":"cad0910a7750f3a086c8e3e80a2947721a5eff857fe607fbebfcd0d0cdb2dde5"}
Sep 09 18:53:46.648810 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:46.648148    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g" event={"ID":"3f9e3ab6-23f7-4e1d-986f-491179ef50dd","Type":"ContainerStarted","Data":"fa0c3830bd0438e74107834dcd6c6d8790dd9cc149f582be919faaa0b0c4290b"}
Sep 09 18:53:46.675978 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:46.675920    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g" podStartSLOduration=1.675903308 podStartE2EDuration="1.675903308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:53:46.673217798 +0000 UTC m=+1314.910587615" watchObservedRunningTime="2026-09-09 18:53:46.675903308 +0000 UTC m=+1314.913273121"
Sep 09 18:53:55.961277 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:55.961233    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:55.961771 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:55.961294    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:55.973938 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:55.973909    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:53:56.726288 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:53:56.726252    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:54:15.573886 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:15.573791    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"]
Sep 09 18:54:15.579234 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:15.579205    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"
Sep 09 18:54:15.582132 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:15.582101    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c09a03\""
Sep 09 18:54:15.588894 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:15.588863    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"]
Sep 09 18:54:15.701989 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:15.701951    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5619f143-51d0-4897-94b9-a7e15c43a57f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94\" (UID: \"5619f143-51d0-4897-94b9-a7e15c43a57f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"
Sep 09 18:54:15.702178 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:15.702052    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l57f8\" (UniqueName: \"kubernetes.io/projected/5619f143-51d0-4897-94b9-a7e15c43a57f-kube-api-access-l57f8\") pod \"payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94\" (UID: \"5619f143-51d0-4897-94b9-a7e15c43a57f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"
Sep 09 18:54:15.802983 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:15.802944    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l57f8\" (UniqueName: \"kubernetes.io/projected/5619f143-51d0-4897-94b9-a7e15c43a57f-kube-api-access-l57f8\") pod \"payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94\" (UID: \"5619f143-51d0-4897-94b9-a7e15c43a57f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"
Sep 09 18:54:15.803176 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:15.803047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5619f143-51d0-4897-94b9-a7e15c43a57f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94\" (UID: \"5619f143-51d0-4897-94b9-a7e15c43a57f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"
Sep 09 18:54:15.803889 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:15.803867    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5619f143-51d0-4897-94b9-a7e15c43a57f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94\" (UID: \"5619f143-51d0-4897-94b9-a7e15c43a57f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"
Sep 09 18:54:15.811666 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:15.811634    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l57f8\" (UniqueName: \"kubernetes.io/projected/5619f143-51d0-4897-94b9-a7e15c43a57f-kube-api-access-l57f8\") pod \"payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94\" (UID: \"5619f143-51d0-4897-94b9-a7e15c43a57f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"
Sep 09 18:54:15.892086 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:15.892029    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"
Sep 09 18:54:16.168594 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:16.167802    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"]
Sep 09 18:54:16.170618 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:54:16.170574    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5619f143_51d0_4897_94b9_a7e15c43a57f.slice/crio-a009376dbcfc4f74fee05dd6874dfd54bc05e02f6ceb2144fa816a1886aebef8 WatchSource:0}: Error finding container a009376dbcfc4f74fee05dd6874dfd54bc05e02f6ceb2144fa816a1886aebef8: Status 404 returned error can't find the container with id a009376dbcfc4f74fee05dd6874dfd54bc05e02f6ceb2144fa816a1886aebef8
Sep 09 18:54:16.820682 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:16.820642    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94" event={"ID":"5619f143-51d0-4897-94b9-a7e15c43a57f","Type":"ContainerStarted","Data":"f8e0e83ae2706aaad7ff36e5fb6b8972fbb9f5cb8d28c8bdf963802ae5ce0424"}
Sep 09 18:54:16.821140 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:16.820700    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94" event={"ID":"5619f143-51d0-4897-94b9-a7e15c43a57f","Type":"ContainerStarted","Data":"a009376dbcfc4f74fee05dd6874dfd54bc05e02f6ceb2144fa816a1886aebef8"}
Sep 09 18:54:16.821140 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:16.820753    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"
Sep 09 18:54:16.840484 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:16.840427    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94" podStartSLOduration=1.840406873 podStartE2EDuration="1.840406873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:54:16.839071576 +0000 UTC m=+1345.076441392" watchObservedRunningTime="2026-09-09 18:54:16.840406873 +0000 UTC m=+1345.077776688"
Sep 09 18:54:27.831096 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:27.831055    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"
Sep 09 18:54:54.348330 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.348295    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"]
Sep 09 18:54:54.352279 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.352255    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"]
Sep 09 18:54:54.352397 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.352366    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"
Sep 09 18:54:54.355705 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.355679    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-212790ce\""
Sep 09 18:54:54.355878 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.355705    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"
Sep 09 18:54:54.364054 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.364023    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"]
Sep 09 18:54:54.378869 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.378834    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"]
Sep 09 18:54:54.391251 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.391214    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h2v2\" (UniqueName: \"kubernetes.io/projected/9df9dc46-31c4-46f0-a2af-b7fdaacc3a09-kube-api-access-7h2v2\") pod \"payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr\" (UID: \"9df9dc46-31c4-46f0-a2af-b7fdaacc3a09\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"
Sep 09 18:54:54.391442 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.391269    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1b0f33-6caa-41b3-b879-ceb7270a1e0b-plugins-config-volume\") pod \"payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt\" (UID: \"da1b0f33-6caa-41b3-b879-ceb7270a1e0b\") " pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"
Sep 09 18:54:54.391442 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.391360    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8hbt\" (UniqueName: \"kubernetes.io/projected/da1b0f33-6caa-41b3-b879-ceb7270a1e0b-kube-api-access-x8hbt\") pod \"payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt\" (UID: \"da1b0f33-6caa-41b3-b879-ceb7270a1e0b\") " pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"
Sep 09 18:54:54.391442 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.391392    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df9dc46-31c4-46f0-a2af-b7fdaacc3a09-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr\" (UID: \"9df9dc46-31c4-46f0-a2af-b7fdaacc3a09\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"
Sep 09 18:54:54.492695 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.492654    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8hbt\" (UniqueName: \"kubernetes.io/projected/da1b0f33-6caa-41b3-b879-ceb7270a1e0b-kube-api-access-x8hbt\") pod \"payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt\" (UID: \"da1b0f33-6caa-41b3-b879-ceb7270a1e0b\") " pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"
Sep 09 18:54:54.492893 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.492707    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df9dc46-31c4-46f0-a2af-b7fdaacc3a09-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr\" (UID: \"9df9dc46-31c4-46f0-a2af-b7fdaacc3a09\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"
Sep 09 18:54:54.492893 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.492863    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h2v2\" (UniqueName: \"kubernetes.io/projected/9df9dc46-31c4-46f0-a2af-b7fdaacc3a09-kube-api-access-7h2v2\") pod \"payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr\" (UID: \"9df9dc46-31c4-46f0-a2af-b7fdaacc3a09\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"
Sep 09 18:54:54.493043 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.492917    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1b0f33-6caa-41b3-b879-ceb7270a1e0b-plugins-config-volume\") pod \"payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt\" (UID: \"da1b0f33-6caa-41b3-b879-ceb7270a1e0b\") " pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"
Sep 09 18:54:54.493595 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.493572    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df9dc46-31c4-46f0-a2af-b7fdaacc3a09-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr\" (UID: \"9df9dc46-31c4-46f0-a2af-b7fdaacc3a09\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"
Sep 09 18:54:54.493595 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.493589    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1b0f33-6caa-41b3-b879-ceb7270a1e0b-plugins-config-volume\") pod \"payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt\" (UID: \"da1b0f33-6caa-41b3-b879-ceb7270a1e0b\") " pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"
Sep 09 18:54:54.502691 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.502659    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h2v2\" (UniqueName: \"kubernetes.io/projected/9df9dc46-31c4-46f0-a2af-b7fdaacc3a09-kube-api-access-7h2v2\") pod \"payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr\" (UID: \"9df9dc46-31c4-46f0-a2af-b7fdaacc3a09\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"
Sep 09 18:54:54.502873 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.502846    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8hbt\" (UniqueName: \"kubernetes.io/projected/da1b0f33-6caa-41b3-b879-ceb7270a1e0b-kube-api-access-x8hbt\") pod \"payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt\" (UID: \"da1b0f33-6caa-41b3-b879-ceb7270a1e0b\") " pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"
Sep 09 18:54:54.665627 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.665584    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"
Sep 09 18:54:54.675129 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.675091    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"
Sep 09 18:54:54.951216 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.951188    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"]
Sep 09 18:54:54.953272 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:54:54.953243    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df9dc46_31c4_46f0_a2af_b7fdaacc3a09.slice/crio-f44a74ab0719efef679a607136bf728a96d279671ed6dbe0cb822102c1066fe8 WatchSource:0}: Error finding container f44a74ab0719efef679a607136bf728a96d279671ed6dbe0cb822102c1066fe8: Status 404 returned error can't find the container with id f44a74ab0719efef679a607136bf728a96d279671ed6dbe0cb822102c1066fe8
Sep 09 18:54:54.970342 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:54.970252    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"]
Sep 09 18:54:54.973372 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:54:54.973339    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1b0f33_6caa_41b3_b879_ceb7270a1e0b.slice/crio-8b83bd0aa306479a0be5d0618c44237106291188d87597ea2abdc7c74da26872 WatchSource:0}: Error finding container 8b83bd0aa306479a0be5d0618c44237106291188d87597ea2abdc7c74da26872: Status 404 returned error can't find the container with id 8b83bd0aa306479a0be5d0618c44237106291188d87597ea2abdc7c74da26872
Sep 09 18:54:55.042125 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:55.042076    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt" event={"ID":"da1b0f33-6caa-41b3-b879-ceb7270a1e0b","Type":"ContainerStarted","Data":"8b83bd0aa306479a0be5d0618c44237106291188d87597ea2abdc7c74da26872"}
Sep 09 18:54:55.044096 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:55.044059    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr" event={"ID":"9df9dc46-31c4-46f0-a2af-b7fdaacc3a09","Type":"ContainerStarted","Data":"9d7da0026876c9353a81cff6a1414f5e4bd81686282e1898d810146db0a2da90"}
Sep 09 18:54:55.044255 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:55.044124    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr" event={"ID":"9df9dc46-31c4-46f0-a2af-b7fdaacc3a09","Type":"ContainerStarted","Data":"f44a74ab0719efef679a607136bf728a96d279671ed6dbe0cb822102c1066fe8"}
Sep 09 18:54:55.044255 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:55.044178    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"
Sep 09 18:54:55.070933 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:55.070863    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr" podStartSLOduration=1.070837917 podStartE2EDuration="1.070837917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:54:55.067969278 +0000 UTC m=+1383.305339094" watchObservedRunningTime="2026-09-09 18:54:55.070837917 +0000 UTC m=+1383.308207733"
Sep 09 18:54:56.054645 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:56.054611    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt" event={"ID":"da1b0f33-6caa-41b3-b879-ceb7270a1e0b","Type":"ContainerStarted","Data":"f038aac90079f48e72c269adcf118f1c030536bc7c3447888280b2756b797f69"}
Sep 09 18:54:56.055088 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:56.054786    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"
Sep 09 18:54:56.074713 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:54:56.074648    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt" podStartSLOduration=2.074624232 podStartE2EDuration="2.074624232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:54:56.072860102 +0000 UTC m=+1384.310229919" watchObservedRunningTime="2026-09-09 18:54:56.074624232 +0000 UTC m=+1384.311994050"
Sep 09 18:55:06.057164 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:06.057124    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"
Sep 09 18:55:07.062702 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:07.062663    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"
Sep 09 18:55:25.038929 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.038890    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"]
Sep 09 18:55:25.042779 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.042751    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.046549 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.046492    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-212790ce\"/\"default-dockercfg-g977n\""
Sep 09 18:55:25.046720 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.046526    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-212790ce\"/\"openshift-service-ca.crt\""
Sep 09 18:55:25.046720 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.046585    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-212790ce\"/\"e2e-lifecycle-model-212790ce-kserve-self-signed-certs\""
Sep 09 18:55:25.046720 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.046631    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-212790ce\"/\"kube-root-ca.crt\""
Sep 09 18:55:25.053220 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.053188    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"]
Sep 09 18:55:25.112082 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.112033    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-dshm\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.112082 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.112079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-home\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.112326 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.112158    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-tls-certs\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.112326 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.112183    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-tmp-dir\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.112326 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.112258    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbp82\" (UniqueName: \"kubernetes.io/projected/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-kube-api-access-mbp82\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.112326 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.112279    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-model-cache\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.213544 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.213508    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-tls-certs\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.213872 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.213554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-tmp-dir\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.213872 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.213672    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbp82\" (UniqueName: \"kubernetes.io/projected/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-kube-api-access-mbp82\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.213872 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.213706    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-model-cache\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.213872 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.213776    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-dshm\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.213872 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.213804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-home\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.214174 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.214103    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-tmp-dir\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.214174 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.214157    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-model-cache\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.214290 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.214177    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-home\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.216195 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.216165    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-dshm\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.216446 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.216429    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-tls-certs\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.223075 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.223044    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbp82\" (UniqueName: \"kubernetes.io/projected/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-kube-api-access-mbp82\") pod \"e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") " pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.354776 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.354662    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:25.634558 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:25.634478    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"]
Sep 09 18:55:25.637400 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:55:25.637370    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d914d2e_efae_4f55_b5b3_fc5f7c184c81.slice/crio-5685b46b476c8752209d463409b68aa7ed4e6d61ecab96efc7f714179ea36d52 WatchSource:0}: Error finding container 5685b46b476c8752209d463409b68aa7ed4e6d61ecab96efc7f714179ea36d52: Status 404 returned error can't find the container with id 5685b46b476c8752209d463409b68aa7ed4e6d61ecab96efc7f714179ea36d52
Sep 09 18:55:26.222874 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:26.222831    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w" event={"ID":"4d914d2e-efae-4f55-b5b3-fc5f7c184c81","Type":"ContainerStarted","Data":"92718ba9501fe24d11ae25839ffa6084fa860359db23bb1d619d324a86256ca8"}
Sep 09 18:55:26.223288 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:26.222888    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w" event={"ID":"4d914d2e-efae-4f55-b5b3-fc5f7c184c81","Type":"ContainerStarted","Data":"5685b46b476c8752209d463409b68aa7ed4e6d61ecab96efc7f714179ea36d52"}
Sep 09 18:55:26.241199 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:26.241137    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w" podStartSLOduration=1.2411164239999999 podStartE2EDuration="1.241116424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:55:26.239343065 +0000 UTC m=+1414.476725456" watchObservedRunningTime="2026-09-09 18:55:26.241116424 +0000 UTC m=+1414.478486238"
Sep 09 18:55:35.355122 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:35.355060    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:35.355122 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:35.355142    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:35.368128 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:35.368096    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:55:36.293010 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:55:36.292967    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:56:06.925815 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:06.925724    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"]
Sep 09 18:56:06.926392 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:06.925986    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt" podUID="da1b0f33-6caa-41b3-b879-ceb7270a1e0b" containerName="payload-processing" containerID="cri-o://f038aac90079f48e72c269adcf118f1c030536bc7c3447888280b2756b797f69" gracePeriod=30
Sep 09 18:56:06.937233 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:06.937201    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"]
Sep 09 18:56:06.937438 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:06.937415    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr" podUID="9df9dc46-31c4-46f0-a2af-b7fdaacc3a09" containerName="payload-pre-processing" containerID="cri-o://9d7da0026876c9353a81cff6a1414f5e4bd81686282e1898d810146db0a2da90" gracePeriod=30
Sep 09 18:56:07.060945 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.060896    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt" podUID="da1b0f33-6caa-41b3-b879-ceb7270a1e0b" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.67:9004: connect: connection refused"
Sep 09 18:56:07.233158 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.233124    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"
Sep 09 18:56:07.236971 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.236943    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"
Sep 09 18:56:07.349131 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.349097    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8hbt\" (UniqueName: \"kubernetes.io/projected/da1b0f33-6caa-41b3-b879-ceb7270a1e0b-kube-api-access-x8hbt\") pod \"da1b0f33-6caa-41b3-b879-ceb7270a1e0b\" (UID: \"da1b0f33-6caa-41b3-b879-ceb7270a1e0b\") "
Sep 09 18:56:07.349131 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.349134    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df9dc46-31c4-46f0-a2af-b7fdaacc3a09-plugins-config-volume\") pod \"9df9dc46-31c4-46f0-a2af-b7fdaacc3a09\" (UID: \"9df9dc46-31c4-46f0-a2af-b7fdaacc3a09\") "
Sep 09 18:56:07.349418 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.349169    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h2v2\" (UniqueName: \"kubernetes.io/projected/9df9dc46-31c4-46f0-a2af-b7fdaacc3a09-kube-api-access-7h2v2\") pod \"9df9dc46-31c4-46f0-a2af-b7fdaacc3a09\" (UID: \"9df9dc46-31c4-46f0-a2af-b7fdaacc3a09\") "
Sep 09 18:56:07.349418 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.349297    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1b0f33-6caa-41b3-b879-ceb7270a1e0b-plugins-config-volume\") pod \"da1b0f33-6caa-41b3-b879-ceb7270a1e0b\" (UID: \"da1b0f33-6caa-41b3-b879-ceb7270a1e0b\") "
Sep 09 18:56:07.349658 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.349533    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df9dc46-31c4-46f0-a2af-b7fdaacc3a09-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9df9dc46-31c4-46f0-a2af-b7fdaacc3a09" (UID: "9df9dc46-31c4-46f0-a2af-b7fdaacc3a09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:56:07.349772 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.349672    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1b0f33-6caa-41b3-b879-ceb7270a1e0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da1b0f33-6caa-41b3-b879-ceb7270a1e0b" (UID: "da1b0f33-6caa-41b3-b879-ceb7270a1e0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:56:07.351367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.351345    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1b0f33-6caa-41b3-b879-ceb7270a1e0b-kube-api-access-x8hbt" (OuterVolumeSpecName: "kube-api-access-x8hbt") pod "da1b0f33-6caa-41b3-b879-ceb7270a1e0b" (UID: "da1b0f33-6caa-41b3-b879-ceb7270a1e0b"). InnerVolumeSpecName "kube-api-access-x8hbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:56:07.351367 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.351355    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df9dc46-31c4-46f0-a2af-b7fdaacc3a09-kube-api-access-7h2v2" (OuterVolumeSpecName: "kube-api-access-7h2v2") pod "9df9dc46-31c4-46f0-a2af-b7fdaacc3a09" (UID: "9df9dc46-31c4-46f0-a2af-b7fdaacc3a09"). InnerVolumeSpecName "kube-api-access-7h2v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:56:07.450597 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.450466    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1b0f33-6caa-41b3-b879-ceb7270a1e0b-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:07.450597 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.450526    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8hbt\" (UniqueName: \"kubernetes.io/projected/da1b0f33-6caa-41b3-b879-ceb7270a1e0b-kube-api-access-x8hbt\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:07.450597 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.450537    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9df9dc46-31c4-46f0-a2af-b7fdaacc3a09-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:07.450597 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.450548    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h2v2\" (UniqueName: \"kubernetes.io/projected/9df9dc46-31c4-46f0-a2af-b7fdaacc3a09-kube-api-access-7h2v2\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:07.453146 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.453113    2575 generic.go:356] "Generic (PLEG): container finished" podID="9df9dc46-31c4-46f0-a2af-b7fdaacc3a09" containerID="9d7da0026876c9353a81cff6a1414f5e4bd81686282e1898d810146db0a2da90" exitCode=0
Sep 09 18:56:07.453316 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.453158    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"
Sep 09 18:56:07.453316 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.453200    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr" event={"ID":"9df9dc46-31c4-46f0-a2af-b7fdaacc3a09","Type":"ContainerDied","Data":"9d7da0026876c9353a81cff6a1414f5e4bd81686282e1898d810146db0a2da90"}
Sep 09 18:56:07.453316 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.453255    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr" event={"ID":"9df9dc46-31c4-46f0-a2af-b7fdaacc3a09","Type":"ContainerDied","Data":"f44a74ab0719efef679a607136bf728a96d279671ed6dbe0cb822102c1066fe8"}
Sep 09 18:56:07.453316 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.453283    2575 scope.go:117] "RemoveContainer" containerID="9d7da0026876c9353a81cff6a1414f5e4bd81686282e1898d810146db0a2da90"
Sep 09 18:56:07.454642 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.454616    2575 generic.go:356] "Generic (PLEG): container finished" podID="da1b0f33-6caa-41b3-b879-ceb7270a1e0b" containerID="f038aac90079f48e72c269adcf118f1c030536bc7c3447888280b2756b797f69" exitCode=0
Sep 09 18:56:07.454746 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.454680    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"
Sep 09 18:56:07.454746 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.454704    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt" event={"ID":"da1b0f33-6caa-41b3-b879-ceb7270a1e0b","Type":"ContainerDied","Data":"f038aac90079f48e72c269adcf118f1c030536bc7c3447888280b2756b797f69"}
Sep 09 18:56:07.454822 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.454749    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt" event={"ID":"da1b0f33-6caa-41b3-b879-ceb7270a1e0b","Type":"ContainerDied","Data":"8b83bd0aa306479a0be5d0618c44237106291188d87597ea2abdc7c74da26872"}
Sep 09 18:56:07.463486 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.463463    2575 scope.go:117] "RemoveContainer" containerID="9d7da0026876c9353a81cff6a1414f5e4bd81686282e1898d810146db0a2da90"
Sep 09 18:56:07.463975 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:56:07.463944    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7da0026876c9353a81cff6a1414f5e4bd81686282e1898d810146db0a2da90\": container with ID starting with 9d7da0026876c9353a81cff6a1414f5e4bd81686282e1898d810146db0a2da90 not found: ID does not exist" containerID="9d7da0026876c9353a81cff6a1414f5e4bd81686282e1898d810146db0a2da90"
Sep 09 18:56:07.464092 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.463987    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7da0026876c9353a81cff6a1414f5e4bd81686282e1898d810146db0a2da90"} err="failed to get container status \"9d7da0026876c9353a81cff6a1414f5e4bd81686282e1898d810146db0a2da90\": rpc error: code = NotFound desc = could not find container \"9d7da0026876c9353a81cff6a1414f5e4bd81686282e1898d810146db0a2da90\": container with ID starting with 9d7da0026876c9353a81cff6a1414f5e4bd81686282e1898d810146db0a2da90 not found: ID does not exist"
Sep 09 18:56:07.464092 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.464011    2575 scope.go:117] "RemoveContainer" containerID="f038aac90079f48e72c269adcf118f1c030536bc7c3447888280b2756b797f69"
Sep 09 18:56:07.474015 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.473995    2575 scope.go:117] "RemoveContainer" containerID="f038aac90079f48e72c269adcf118f1c030536bc7c3447888280b2756b797f69"
Sep 09 18:56:07.474374 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:56:07.474353    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f038aac90079f48e72c269adcf118f1c030536bc7c3447888280b2756b797f69\": container with ID starting with f038aac90079f48e72c269adcf118f1c030536bc7c3447888280b2756b797f69 not found: ID does not exist" containerID="f038aac90079f48e72c269adcf118f1c030536bc7c3447888280b2756b797f69"
Sep 09 18:56:07.474424 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.474385    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f038aac90079f48e72c269adcf118f1c030536bc7c3447888280b2756b797f69"} err="failed to get container status \"f038aac90079f48e72c269adcf118f1c030536bc7c3447888280b2756b797f69\": rpc error: code = NotFound desc = could not find container \"f038aac90079f48e72c269adcf118f1c030536bc7c3447888280b2756b797f69\": container with ID starting with f038aac90079f48e72c269adcf118f1c030536bc7c3447888280b2756b797f69 not found: ID does not exist"
Sep 09 18:56:07.484064 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.484022    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"]
Sep 09 18:56:07.493276 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.493237    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-212790ce-545dbf966-vrqdr"]
Sep 09 18:56:07.505862 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.505822    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"]
Sep 09 18:56:07.512153 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:07.512121    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-212790ce-7d5d67f68f-klflt"]
Sep 09 18:56:08.322152 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:08.322066    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9df9dc46-31c4-46f0-a2af-b7fdaacc3a09" path="/var/lib/kubelet/pods/9df9dc46-31c4-46f0-a2af-b7fdaacc3a09/volumes"
Sep 09 18:56:08.322552 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:08.322457    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da1b0f33-6caa-41b3-b879-ceb7270a1e0b" path="/var/lib/kubelet/pods/da1b0f33-6caa-41b3-b879-ceb7270a1e0b/volumes"
Sep 09 18:56:10.683567 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:10.683532    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"]
Sep 09 18:56:11.734345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:11.734304    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"]
Sep 09 18:56:15.522384 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:15.522346    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"]
Sep 09 18:56:25.704161 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:25.704098    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l" podUID="4095c852-bb8d-4cfa-97e4-d90ae2e385ac" containerName="main" containerID="cri-o://09c9c86902a2e30ec731e073e3505cb01ac38d1660a48f414f55311fd63bcaf5" gracePeriod=45
Sep 09 18:56:25.963952 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:25.963866    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:56:26.049396 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.049360    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-tls-certs\") pod \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") "
Sep 09 18:56:26.049396 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.049402    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-dshm\") pod \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") "
Sep 09 18:56:26.049708 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.049417    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-tmp-dir\") pod \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") "
Sep 09 18:56:26.049708 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.049456    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-model-cache\") pod \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") "
Sep 09 18:56:26.049708 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.049481    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-home\") pod \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") "
Sep 09 18:56:26.049708 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.049530    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x2ms\" (UniqueName: \"kubernetes.io/projected/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-kube-api-access-9x2ms\") pod \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\" (UID: \"4095c852-bb8d-4cfa-97e4-d90ae2e385ac\") "
Sep 09 18:56:26.049901 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.049766    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4095c852-bb8d-4cfa-97e4-d90ae2e385ac" (UID: "4095c852-bb8d-4cfa-97e4-d90ae2e385ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:56:26.049901 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.049777    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "4095c852-bb8d-4cfa-97e4-d90ae2e385ac" (UID: "4095c852-bb8d-4cfa-97e4-d90ae2e385ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:56:26.049901 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.049787    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-home" (OuterVolumeSpecName: "home") pod "4095c852-bb8d-4cfa-97e4-d90ae2e385ac" (UID: "4095c852-bb8d-4cfa-97e4-d90ae2e385ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:56:26.050006 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.049953    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:26.050006 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.049970    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:26.050006 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.049985    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:26.051912 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.051887    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4095c852-bb8d-4cfa-97e4-d90ae2e385ac" (UID: "4095c852-bb8d-4cfa-97e4-d90ae2e385ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:56:26.051996 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.051890    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-dshm" (OuterVolumeSpecName: "dshm") pod "4095c852-bb8d-4cfa-97e4-d90ae2e385ac" (UID: "4095c852-bb8d-4cfa-97e4-d90ae2e385ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:56:26.051996 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.051974    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-kube-api-access-9x2ms" (OuterVolumeSpecName: "kube-api-access-9x2ms") pod "4095c852-bb8d-4cfa-97e4-d90ae2e385ac" (UID: "4095c852-bb8d-4cfa-97e4-d90ae2e385ac"). InnerVolumeSpecName "kube-api-access-9x2ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:56:26.150741 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.150693    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:26.150741 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.150733    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:26.150741 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.150743    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x2ms\" (UniqueName: \"kubernetes.io/projected/4095c852-bb8d-4cfa-97e4-d90ae2e385ac-kube-api-access-9x2ms\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:26.568743 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.568696    2575 generic.go:356] "Generic (PLEG): container finished" podID="4095c852-bb8d-4cfa-97e4-d90ae2e385ac" containerID="09c9c86902a2e30ec731e073e3505cb01ac38d1660a48f414f55311fd63bcaf5" exitCode=0
Sep 09 18:56:26.568912 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.568748    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"
Sep 09 18:56:26.568912 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.568746    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l" event={"ID":"4095c852-bb8d-4cfa-97e4-d90ae2e385ac","Type":"ContainerDied","Data":"09c9c86902a2e30ec731e073e3505cb01ac38d1660a48f414f55311fd63bcaf5"}
Sep 09 18:56:26.568912 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.568869    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l" event={"ID":"4095c852-bb8d-4cfa-97e4-d90ae2e385ac","Type":"ContainerDied","Data":"628911873e2a1f6b7a8a646c0205da039516649745a75a119eadd9c1dbc931e2"}
Sep 09 18:56:26.568912 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.568890    2575 scope.go:117] "RemoveContainer" containerID="09c9c86902a2e30ec731e073e3505cb01ac38d1660a48f414f55311fd63bcaf5"
Sep 09 18:56:26.578283 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.578257    2575 scope.go:117] "RemoveContainer" containerID="09c9c86902a2e30ec731e073e3505cb01ac38d1660a48f414f55311fd63bcaf5"
Sep 09 18:56:26.578723 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:56:26.578655    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c9c86902a2e30ec731e073e3505cb01ac38d1660a48f414f55311fd63bcaf5\": container with ID starting with 09c9c86902a2e30ec731e073e3505cb01ac38d1660a48f414f55311fd63bcaf5 not found: ID does not exist" containerID="09c9c86902a2e30ec731e073e3505cb01ac38d1660a48f414f55311fd63bcaf5"
Sep 09 18:56:26.578723 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.578707    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c9c86902a2e30ec731e073e3505cb01ac38d1660a48f414f55311fd63bcaf5"} err="failed to get container status \"09c9c86902a2e30ec731e073e3505cb01ac38d1660a48f414f55311fd63bcaf5\": rpc error: code = NotFound desc = could not find container \"09c9c86902a2e30ec731e073e3505cb01ac38d1660a48f414f55311fd63bcaf5\": container with ID starting with 09c9c86902a2e30ec731e073e3505cb01ac38d1660a48f414f55311fd63bcaf5 not found: ID does not exist"
Sep 09 18:56:26.586354 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.586322    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"]
Sep 09 18:56:26.592438 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.592405    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/auth-test-model-282de3-kserve-5987d676bc-8l52l"]
Sep 09 18:56:26.755200 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:26.755158    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g" podUID="3f9e3ab6-23f7-4e1d-986f-491179ef50dd" containerName="main" containerID="cri-o://cad0910a7750f3a086c8e3e80a2947721a5eff857fe607fbebfcd0d0cdb2dde5" gracePeriod=45
Sep 09 18:56:27.009654 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.009622    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:56:27.160685 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.160578    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-dshm\") pod \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") "
Sep 09 18:56:27.160685 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.160645    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-tmp-dir\") pod \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") "
Sep 09 18:56:27.160883 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.160747    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-tls-certs\") pod \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") "
Sep 09 18:56:27.160883 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.160787    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-model-cache\") pod \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") "
Sep 09 18:56:27.160883 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.160845    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbqj8\" (UniqueName: \"kubernetes.io/projected/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-kube-api-access-mbqj8\") pod \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") "
Sep 09 18:56:27.161012 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.160888    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-home\") pod \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\" (UID: \"3f9e3ab6-23f7-4e1d-986f-491179ef50dd\") "
Sep 09 18:56:27.161012 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.160966    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f9e3ab6-23f7-4e1d-986f-491179ef50dd" (UID: "3f9e3ab6-23f7-4e1d-986f-491179ef50dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:56:27.161123 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.161082    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f9e3ab6-23f7-4e1d-986f-491179ef50dd" (UID: "3f9e3ab6-23f7-4e1d-986f-491179ef50dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:56:27.161192 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.161173    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-home" (OuterVolumeSpecName: "home") pod "3f9e3ab6-23f7-4e1d-986f-491179ef50dd" (UID: "3f9e3ab6-23f7-4e1d-986f-491179ef50dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:56:27.161281 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.161266    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:27.161336 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.161283    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:27.161336 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.161292    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:27.163045 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.163018    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-dshm" (OuterVolumeSpecName: "dshm") pod "3f9e3ab6-23f7-4e1d-986f-491179ef50dd" (UID: "3f9e3ab6-23f7-4e1d-986f-491179ef50dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:56:27.163166 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.163118    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-kube-api-access-mbqj8" (OuterVolumeSpecName: "kube-api-access-mbqj8") pod "3f9e3ab6-23f7-4e1d-986f-491179ef50dd" (UID: "3f9e3ab6-23f7-4e1d-986f-491179ef50dd"). InnerVolumeSpecName "kube-api-access-mbqj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:56:27.163166 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.163125    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f9e3ab6-23f7-4e1d-986f-491179ef50dd" (UID: "3f9e3ab6-23f7-4e1d-986f-491179ef50dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:56:27.262654 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.262615    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:27.262654 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.262654    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbqj8\" (UniqueName: \"kubernetes.io/projected/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-kube-api-access-mbqj8\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:27.262878 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.262667    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f9e3ab6-23f7-4e1d-986f-491179ef50dd-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:27.584896 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.584798    2575 generic.go:356] "Generic (PLEG): container finished" podID="3f9e3ab6-23f7-4e1d-986f-491179ef50dd" containerID="cad0910a7750f3a086c8e3e80a2947721a5eff857fe607fbebfcd0d0cdb2dde5" exitCode=0
Sep 09 18:56:27.584896 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.584847    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g" event={"ID":"3f9e3ab6-23f7-4e1d-986f-491179ef50dd","Type":"ContainerDied","Data":"cad0910a7750f3a086c8e3e80a2947721a5eff857fe607fbebfcd0d0cdb2dde5"}
Sep 09 18:56:27.584896 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.584856    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"
Sep 09 18:56:27.584896 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.584880    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g" event={"ID":"3f9e3ab6-23f7-4e1d-986f-491179ef50dd","Type":"ContainerDied","Data":"fa0c3830bd0438e74107834dcd6c6d8790dd9cc149f582be919faaa0b0c4290b"}
Sep 09 18:56:27.584896 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.584899    2575 scope.go:117] "RemoveContainer" containerID="cad0910a7750f3a086c8e3e80a2947721a5eff857fe607fbebfcd0d0cdb2dde5"
Sep 09 18:56:27.594868 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.594848    2575 scope.go:117] "RemoveContainer" containerID="cad0910a7750f3a086c8e3e80a2947721a5eff857fe607fbebfcd0d0cdb2dde5"
Sep 09 18:56:27.595194 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:56:27.595173    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad0910a7750f3a086c8e3e80a2947721a5eff857fe607fbebfcd0d0cdb2dde5\": container with ID starting with cad0910a7750f3a086c8e3e80a2947721a5eff857fe607fbebfcd0d0cdb2dde5 not found: ID does not exist" containerID="cad0910a7750f3a086c8e3e80a2947721a5eff857fe607fbebfcd0d0cdb2dde5"
Sep 09 18:56:27.595245 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.595204    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad0910a7750f3a086c8e3e80a2947721a5eff857fe607fbebfcd0d0cdb2dde5"} err="failed to get container status \"cad0910a7750f3a086c8e3e80a2947721a5eff857fe607fbebfcd0d0cdb2dde5\": rpc error: code = NotFound desc = could not find container \"cad0910a7750f3a086c8e3e80a2947721a5eff857fe607fbebfcd0d0cdb2dde5\": container with ID starting with cad0910a7750f3a086c8e3e80a2947721a5eff857fe607fbebfcd0d0cdb2dde5 not found: ID does not exist"
Sep 09 18:56:27.606873 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.606833    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"]
Sep 09 18:56:27.610190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:27.610151    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/auth-test-model-0e012d-kserve-864c5968f9-xdt4g"]
Sep 09 18:56:28.322024 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:28.321987    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f9e3ab6-23f7-4e1d-986f-491179ef50dd" path="/var/lib/kubelet/pods/3f9e3ab6-23f7-4e1d-986f-491179ef50dd/volumes"
Sep 09 18:56:28.322443 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:28.322402    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4095c852-bb8d-4cfa-97e4-d90ae2e385ac" path="/var/lib/kubelet/pods/4095c852-bb8d-4cfa-97e4-d90ae2e385ac/volumes"
Sep 09 18:56:30.542781 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.542721    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w" podUID="4d914d2e-efae-4f55-b5b3-fc5f7c184c81" containerName="main" containerID="cri-o://92718ba9501fe24d11ae25839ffa6084fa860359db23bb1d619d324a86256ca8" gracePeriod=45
Sep 09 18:56:30.809015 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.808936    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:56:30.900396 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.900359    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbp82\" (UniqueName: \"kubernetes.io/projected/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-kube-api-access-mbp82\") pod \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") "
Sep 09 18:56:30.900619 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.900410    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-tls-certs\") pod \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") "
Sep 09 18:56:30.900619 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.900467    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-home\") pod \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") "
Sep 09 18:56:30.900619 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.900547    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-model-cache\") pod \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") "
Sep 09 18:56:30.900619 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.900596    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-dshm\") pod \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") "
Sep 09 18:56:30.900839 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.900656    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-tmp-dir\") pod \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\" (UID: \"4d914d2e-efae-4f55-b5b3-fc5f7c184c81\") "
Sep 09 18:56:30.900839 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.900784    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-home" (OuterVolumeSpecName: "home") pod "4d914d2e-efae-4f55-b5b3-fc5f7c184c81" (UID: "4d914d2e-efae-4f55-b5b3-fc5f7c184c81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:56:30.900927 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.900895    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d914d2e-efae-4f55-b5b3-fc5f7c184c81" (UID: "4d914d2e-efae-4f55-b5b3-fc5f7c184c81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:56:30.900973 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.900933    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d914d2e-efae-4f55-b5b3-fc5f7c184c81" (UID: "4d914d2e-efae-4f55-b5b3-fc5f7c184c81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:56:30.901047 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.901029    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:30.901085 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.901056    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:30.901130 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.901089    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:30.902757 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.902736    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-kube-api-access-mbp82" (OuterVolumeSpecName: "kube-api-access-mbp82") pod "4d914d2e-efae-4f55-b5b3-fc5f7c184c81" (UID: "4d914d2e-efae-4f55-b5b3-fc5f7c184c81"). InnerVolumeSpecName "kube-api-access-mbp82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:56:30.903037 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.903007    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d914d2e-efae-4f55-b5b3-fc5f7c184c81" (UID: "4d914d2e-efae-4f55-b5b3-fc5f7c184c81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:56:30.903037 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:30.903019    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-dshm" (OuterVolumeSpecName: "dshm") pod "4d914d2e-efae-4f55-b5b3-fc5f7c184c81" (UID: "4d914d2e-efae-4f55-b5b3-fc5f7c184c81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:56:31.002171 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:31.002120    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbp82\" (UniqueName: \"kubernetes.io/projected/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-kube-api-access-mbp82\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:31.002171 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:31.002161    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:31.002171 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:31.002171    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d914d2e-efae-4f55-b5b3-fc5f7c184c81-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:56:31.609745 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:31.609711    2575 generic.go:356] "Generic (PLEG): container finished" podID="4d914d2e-efae-4f55-b5b3-fc5f7c184c81" containerID="92718ba9501fe24d11ae25839ffa6084fa860359db23bb1d619d324a86256ca8" exitCode=0
Sep 09 18:56:31.610238 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:31.609769    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"
Sep 09 18:56:31.610238 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:31.609798    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w" event={"ID":"4d914d2e-efae-4f55-b5b3-fc5f7c184c81","Type":"ContainerDied","Data":"92718ba9501fe24d11ae25839ffa6084fa860359db23bb1d619d324a86256ca8"}
Sep 09 18:56:31.610238 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:31.609845    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w" event={"ID":"4d914d2e-efae-4f55-b5b3-fc5f7c184c81","Type":"ContainerDied","Data":"5685b46b476c8752209d463409b68aa7ed4e6d61ecab96efc7f714179ea36d52"}
Sep 09 18:56:31.610238 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:31.609865    2575 scope.go:117] "RemoveContainer" containerID="92718ba9501fe24d11ae25839ffa6084fa860359db23bb1d619d324a86256ca8"
Sep 09 18:56:31.619574 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:31.619552    2575 scope.go:117] "RemoveContainer" containerID="92718ba9501fe24d11ae25839ffa6084fa860359db23bb1d619d324a86256ca8"
Sep 09 18:56:31.619948 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:56:31.619928    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92718ba9501fe24d11ae25839ffa6084fa860359db23bb1d619d324a86256ca8\": container with ID starting with 92718ba9501fe24d11ae25839ffa6084fa860359db23bb1d619d324a86256ca8 not found: ID does not exist" containerID="92718ba9501fe24d11ae25839ffa6084fa860359db23bb1d619d324a86256ca8"
Sep 09 18:56:31.620018 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:31.619957    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92718ba9501fe24d11ae25839ffa6084fa860359db23bb1d619d324a86256ca8"} err="failed to get container status \"92718ba9501fe24d11ae25839ffa6084fa860359db23bb1d619d324a86256ca8\": rpc error: code = NotFound desc = could not find container \"92718ba9501fe24d11ae25839ffa6084fa860359db23bb1d619d324a86256ca8\": container with ID starting with 92718ba9501fe24d11ae25839ffa6084fa860359db23bb1d619d324a86256ca8 not found: ID does not exist"
Sep 09 18:56:31.632169 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:31.632132    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"]
Sep 09 18:56:31.636777 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:31.636745    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-212790ce/e2e-lifecycle-model-212790ce-kserve-7d87bf75f-l5s6w"]
Sep 09 18:56:32.321300 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.321261    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d914d2e-efae-4f55-b5b3-fc5f7c184c81" path="/var/lib/kubelet/pods/4d914d2e-efae-4f55-b5b3-fc5f7c184c81/volumes"
Sep 09 18:56:32.746276 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746240    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"]
Sep 09 18:56:32.746729 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746712    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df9dc46-31c4-46f0-a2af-b7fdaacc3a09" containerName="payload-pre-processing"
Sep 09 18:56:32.746789 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746731    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df9dc46-31c4-46f0-a2af-b7fdaacc3a09" containerName="payload-pre-processing"
Sep 09 18:56:32.746789 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746742    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4095c852-bb8d-4cfa-97e4-d90ae2e385ac" containerName="main"
Sep 09 18:56:32.746789 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746749    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4095c852-bb8d-4cfa-97e4-d90ae2e385ac" containerName="main"
Sep 09 18:56:32.746789 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746756    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1b0f33-6caa-41b3-b879-ceb7270a1e0b" containerName="payload-processing"
Sep 09 18:56:32.746789 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746763    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1b0f33-6caa-41b3-b879-ceb7270a1e0b" containerName="payload-processing"
Sep 09 18:56:32.746789 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746778    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f9e3ab6-23f7-4e1d-986f-491179ef50dd" containerName="main"
Sep 09 18:56:32.746789 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746783    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f9e3ab6-23f7-4e1d-986f-491179ef50dd" containerName="main"
Sep 09 18:56:32.746789 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746789    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d914d2e-efae-4f55-b5b3-fc5f7c184c81" containerName="main"
Sep 09 18:56:32.747038 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746797    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d914d2e-efae-4f55-b5b3-fc5f7c184c81" containerName="main"
Sep 09 18:56:32.747038 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746865    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4095c852-bb8d-4cfa-97e4-d90ae2e385ac" containerName="main"
Sep 09 18:56:32.747038 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746876    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d914d2e-efae-4f55-b5b3-fc5f7c184c81" containerName="main"
Sep 09 18:56:32.747038 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746886    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f9e3ab6-23f7-4e1d-986f-491179ef50dd" containerName="main"
Sep 09 18:56:32.747038 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746892    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="da1b0f33-6caa-41b3-b879-ceb7270a1e0b" containerName="payload-processing"
Sep 09 18:56:32.747038 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.746900    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df9dc46-31c4-46f0-a2af-b7fdaacc3a09" containerName="payload-pre-processing"
Sep 09 18:56:32.757338 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.757302    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.760894 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.760860    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-282de3\"/\"openshift-service-ca.crt\""
Sep 09 18:56:32.762143 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.762116    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-282de3\"/\"sub-test-model-282de3-kserve-self-signed-certs\""
Sep 09 18:56:32.762728 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.762700    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-282de3\"/\"kube-root-ca.crt\""
Sep 09 18:56:32.762993 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.762974    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"]
Sep 09 18:56:32.763076 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.763044    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-282de3\"/\"default-dockercfg-bjp96\""
Sep 09 18:56:32.816114 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.816074    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d84dd66-04b3-4254-8e6e-bbb464b1866c-tls-certs\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.816114 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.816116    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-model-cache\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.816366 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.816195    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bbps\" (UniqueName: \"kubernetes.io/projected/5d84dd66-04b3-4254-8e6e-bbb464b1866c-kube-api-access-6bbps\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.816366 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.816244    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-home\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.816366 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.816316    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-tmp-dir\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.816366 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.816355    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-dshm\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.917363 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.917314    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-tmp-dir\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.917363 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.917367    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-dshm\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.917647 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.917415    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d84dd66-04b3-4254-8e6e-bbb464b1866c-tls-certs\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.917647 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.917433    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-model-cache\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.917647 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.917467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bbps\" (UniqueName: \"kubernetes.io/projected/5d84dd66-04b3-4254-8e6e-bbb464b1866c-kube-api-access-6bbps\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.917647 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.917565    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-home\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.917862 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.917824    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-tmp-dir\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.917928 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.917900    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-model-cache\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.917998 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.917978    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-home\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.919877 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.919855    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-dshm\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.920130 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.920113    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d84dd66-04b3-4254-8e6e-bbb464b1866c-tls-certs\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:32.926114 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:32.926088    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bbps\" (UniqueName: \"kubernetes.io/projected/5d84dd66-04b3-4254-8e6e-bbb464b1866c-kube-api-access-6bbps\") pod \"sub-test-model-282de3-kserve-56cb66f9-dkspd\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") " pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:33.070712 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:33.070596    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:33.351694 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:33.351664    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"]
Sep 09 18:56:33.354378 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:56:33.354340    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d84dd66_04b3_4254_8e6e_bbb464b1866c.slice/crio-db9a860aea5f0b331bcf1c2c5bf4a6efb4e81ec94cb38fc70d071233f17f3e02 WatchSource:0}: Error finding container db9a860aea5f0b331bcf1c2c5bf4a6efb4e81ec94cb38fc70d071233f17f3e02: Status 404 returned error can't find the container with id db9a860aea5f0b331bcf1c2c5bf4a6efb4e81ec94cb38fc70d071233f17f3e02
Sep 09 18:56:33.623317 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:33.623277    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd" event={"ID":"5d84dd66-04b3-4254-8e6e-bbb464b1866c","Type":"ContainerStarted","Data":"aa726ef4225b1d3bb3475dfcb395a3b6fc5f51188b8d3fc6e7d0e5fe77385a15"}
Sep 09 18:56:33.623317 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:33.623320    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd" event={"ID":"5d84dd66-04b3-4254-8e6e-bbb464b1866c","Type":"ContainerStarted","Data":"db9a860aea5f0b331bcf1c2c5bf4a6efb4e81ec94cb38fc70d071233f17f3e02"}
Sep 09 18:56:33.642325 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:33.642266    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd" podStartSLOduration=1.642249073 podStartE2EDuration="1.642249073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:56:33.641737034 +0000 UTC m=+1481.879106849" watchObservedRunningTime="2026-09-09 18:56:33.642249073 +0000 UTC m=+1481.879618888"
Sep 09 18:56:43.070970 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:43.070907    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:43.071585 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:43.070997    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:43.084173 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:43.084136    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:43.691218 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:43.691179    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:56:52.339626 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:52.339594    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 18:56:52.343130 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:56:52.343105    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 18:57:31.435625 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.435592    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"]
Sep 09 18:57:31.439175 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.439155    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.442933 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.442905    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e012d\"/\"kube-root-ca.crt\""
Sep 09 18:57:31.442933 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.442929    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e012d\"/\"openshift-service-ca.crt\""
Sep 09 18:57:31.443155 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.442955    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e012d\"/\"sub-test-model-0e012d-kserve-self-signed-certs\""
Sep 09 18:57:31.443155 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.442908    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e012d\"/\"default-dockercfg-6kjms\""
Sep 09 18:57:31.449790 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.449761    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"]
Sep 09 18:57:31.528580 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.528532    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjb8g\" (UniqueName: \"kubernetes.io/projected/945f5039-d593-4225-a050-fb4d9eb54c5e-kube-api-access-rjb8g\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.528813 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.528598    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-tmp-dir\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.528813 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.528669    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-model-cache\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.528813 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.528754    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-dshm\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.528813 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.528797    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-home\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.529024 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.528861    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/945f5039-d593-4225-a050-fb4d9eb54c5e-tls-certs\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.629890 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.629844    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-tmp-dir\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.630071 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.629946    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-model-cache\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.630071 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.629978    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-dshm\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.630071 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.629996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-home\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.630210 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.630143    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/945f5039-d593-4225-a050-fb4d9eb54c5e-tls-certs\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.630267 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.630221    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjb8g\" (UniqueName: \"kubernetes.io/projected/945f5039-d593-4225-a050-fb4d9eb54c5e-kube-api-access-rjb8g\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.630321 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.630297    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-tmp-dir\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.630390 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.630371    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-model-cache\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.630454 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.630416    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-home\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.632299 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.632275    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-dshm\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.632788 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.632767    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/945f5039-d593-4225-a050-fb4d9eb54c5e-tls-certs\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.639014 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.638977    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjb8g\" (UniqueName: \"kubernetes.io/projected/945f5039-d593-4225-a050-fb4d9eb54c5e-kube-api-access-rjb8g\") pod \"sub-test-model-0e012d-kserve-6f8f86c795-f7zmh\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:31.750282 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:31.750181    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:32.027190 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:32.027157    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"]
Sep 09 18:57:32.029451 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:57:32.029415    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod945f5039_d593_4225_a050_fb4d9eb54c5e.slice/crio-bf0889e4b02641479ed3d5079492be67e4c3c37bde3c334dc0b8b3996e870d39 WatchSource:0}: Error finding container bf0889e4b02641479ed3d5079492be67e4c3c37bde3c334dc0b8b3996e870d39: Status 404 returned error can't find the container with id bf0889e4b02641479ed3d5079492be67e4c3c37bde3c334dc0b8b3996e870d39
Sep 09 18:57:32.951586 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:32.951545    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh" event={"ID":"945f5039-d593-4225-a050-fb4d9eb54c5e","Type":"ContainerStarted","Data":"62f581ec92d61643e21617239a08bccba5b968e3c9b233f730388742f5b36e3b"}
Sep 09 18:57:32.951586 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:32.951594    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh" event={"ID":"945f5039-d593-4225-a050-fb4d9eb54c5e","Type":"ContainerStarted","Data":"bf0889e4b02641479ed3d5079492be67e4c3c37bde3c334dc0b8b3996e870d39"}
Sep 09 18:57:32.967615 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:32.967553    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh" podStartSLOduration=1.967537182 podStartE2EDuration="1.967537182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:57:32.966682692 +0000 UTC m=+1541.204052525" watchObservedRunningTime="2026-09-09 18:57:32.967537182 +0000 UTC m=+1541.204907038"
Sep 09 18:57:41.750860 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:41.750816    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:41.751290 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:41.750878    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:41.764016 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:41.763982    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:42.012168 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:42.012082    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:57:58.083973 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.083933    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"]
Sep 09 18:57:58.087705 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.087679    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"
Sep 09 18:57:58.090584 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.090558    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-79b1ac\""
Sep 09 18:57:58.090771 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.090684    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-79b1ac-dockercfg-tcklc\""
Sep 09 18:57:58.101699 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.101665    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"]
Sep 09 18:57:58.131035 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.130994    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"]
Sep 09 18:57:58.134535 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.134513    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"
Sep 09 18:57:58.140272 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.140243    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"]
Sep 09 18:57:58.209248 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.209210    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqllf\" (UniqueName: \"kubernetes.io/projected/83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7-kube-api-access-qqllf\") pod \"payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2\" (UID: \"83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"
Sep 09 18:57:58.209450 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.209308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7beb3fb8-51b9-46ba-be48-ed5f50533aef-plugins-config-volume\") pod \"payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762\" (UID: \"7beb3fb8-51b9-46ba-be48-ed5f50533aef\") " pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"
Sep 09 18:57:58.209450 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.209341    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2\" (UID: \"83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"
Sep 09 18:57:58.209450 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.209402    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rwd\" (UniqueName: \"kubernetes.io/projected/7beb3fb8-51b9-46ba-be48-ed5f50533aef-kube-api-access-52rwd\") pod \"payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762\" (UID: \"7beb3fb8-51b9-46ba-be48-ed5f50533aef\") " pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"
Sep 09 18:57:58.310959 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.310907    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7beb3fb8-51b9-46ba-be48-ed5f50533aef-plugins-config-volume\") pod \"payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762\" (UID: \"7beb3fb8-51b9-46ba-be48-ed5f50533aef\") " pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"
Sep 09 18:57:58.311183 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.310970    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2\" (UID: \"83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"
Sep 09 18:57:58.311183 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.311060    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rwd\" (UniqueName: \"kubernetes.io/projected/7beb3fb8-51b9-46ba-be48-ed5f50533aef-kube-api-access-52rwd\") pod \"payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762\" (UID: \"7beb3fb8-51b9-46ba-be48-ed5f50533aef\") " pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"
Sep 09 18:57:58.311183 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.311112    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqllf\" (UniqueName: \"kubernetes.io/projected/83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7-kube-api-access-qqllf\") pod \"payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2\" (UID: \"83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"
Sep 09 18:57:58.311783 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.311761    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7beb3fb8-51b9-46ba-be48-ed5f50533aef-plugins-config-volume\") pod \"payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762\" (UID: \"7beb3fb8-51b9-46ba-be48-ed5f50533aef\") " pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"
Sep 09 18:57:58.311850 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.311789    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2\" (UID: \"83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"
Sep 09 18:57:58.319471 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.319443    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqllf\" (UniqueName: \"kubernetes.io/projected/83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7-kube-api-access-qqllf\") pod \"payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2\" (UID: \"83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"
Sep 09 18:57:58.319750 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.319720    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rwd\" (UniqueName: \"kubernetes.io/projected/7beb3fb8-51b9-46ba-be48-ed5f50533aef-kube-api-access-52rwd\") pod \"payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762\" (UID: \"7beb3fb8-51b9-46ba-be48-ed5f50533aef\") " pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"
Sep 09 18:57:58.398641 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.398599    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"
Sep 09 18:57:58.446882 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.446837    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"
Sep 09 18:57:58.682949 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.682921    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"]
Sep 09 18:57:58.685306 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:57:58.685270    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83db1a1e_e4f3_4ef0_ba09_e55adf1aefc7.slice/crio-c9e036d7f4d40bbd98372223c7df1da67dc988c1e6cc875c2b8ddabf8f76764e WatchSource:0}: Error finding container c9e036d7f4d40bbd98372223c7df1da67dc988c1e6cc875c2b8ddabf8f76764e: Status 404 returned error can't find the container with id c9e036d7f4d40bbd98372223c7df1da67dc988c1e6cc875c2b8ddabf8f76764e
Sep 09 18:57:58.734881 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:58.734849    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"]
Sep 09 18:57:58.737649 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:57:58.737617    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7beb3fb8_51b9_46ba_be48_ed5f50533aef.slice/crio-c011e5177f1b39553ed0e16347565ebd465ab7d60fd4fcf3c9d64c36e2a89047 WatchSource:0}: Error finding container c011e5177f1b39553ed0e16347565ebd465ab7d60fd4fcf3c9d64c36e2a89047: Status 404 returned error can't find the container with id c011e5177f1b39553ed0e16347565ebd465ab7d60fd4fcf3c9d64c36e2a89047
Sep 09 18:57:59.098994 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:59.098883    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762" event={"ID":"7beb3fb8-51b9-46ba-be48-ed5f50533aef","Type":"ContainerStarted","Data":"ef2dacb16b7f86dac2f5fee87dda3cad30cf56893eeebf1732282799fb1db1e8"}
Sep 09 18:57:59.098994 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:59.098949    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762" event={"ID":"7beb3fb8-51b9-46ba-be48-ed5f50533aef","Type":"ContainerStarted","Data":"c011e5177f1b39553ed0e16347565ebd465ab7d60fd4fcf3c9d64c36e2a89047"}
Sep 09 18:57:59.099558 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:59.099091    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"
Sep 09 18:57:59.100793 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:59.100760    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2" event={"ID":"83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7","Type":"ContainerStarted","Data":"49a2400627fe86de7720063d556a378e598d41f4cbc9017de509919c05cb7f37"}
Sep 09 18:57:59.100937 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:59.100816    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2" event={"ID":"83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7","Type":"ContainerStarted","Data":"c9e036d7f4d40bbd98372223c7df1da67dc988c1e6cc875c2b8ddabf8f76764e"}
Sep 09 18:57:59.100937 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:59.100888    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"
Sep 09 18:57:59.116980 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:59.116926    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762" podStartSLOduration=1.116909645 podStartE2EDuration="1.116909645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:57:59.115617641 +0000 UTC m=+1567.352987457" watchObservedRunningTime="2026-09-09 18:57:59.116909645 +0000 UTC m=+1567.354279459"
Sep 09 18:57:59.132975 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:57:59.132917    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2" podStartSLOduration=1.1328984229999999 podStartE2EDuration="1.132898423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:57:59.130454557 +0000 UTC m=+1567.367824363" watchObservedRunningTime="2026-09-09 18:57:59.132898423 +0000 UTC m=+1567.370268289"
Sep 09 18:58:10.109534 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:10.109477    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"
Sep 09 18:58:10.110039 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:10.109661    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"
Sep 09 18:58:17.236174 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:17.236139    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"]
Sep 09 18:58:18.033019 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:18.032976    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"]
Sep 09 18:58:27.668730 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.668689    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"]
Sep 09 18:58:27.672456 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.672430    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"
Sep 09 18:58:27.675766 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.675730    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8fbc9c\""
Sep 09 18:58:27.685452 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.685417    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"]
Sep 09 18:58:27.696026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.695986    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"]
Sep 09 18:58:27.700225 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.700198    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"
Sep 09 18:58:27.702760 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.702727    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8fbc9c-dockercfg-8x6f2\""
Sep 09 18:58:27.712433 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.712403    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"]
Sep 09 18:58:27.712632 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.712606    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62185ffe-3de7-4b50-bcb0-cb23fb5abcd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424\" (UID: \"62185ffe-3de7-4b50-bcb0-cb23fb5abcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"
Sep 09 18:58:27.712691 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.712671    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxlhx\" (UniqueName: \"kubernetes.io/projected/62185ffe-3de7-4b50-bcb0-cb23fb5abcd9-kube-api-access-bxlhx\") pod \"payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424\" (UID: \"62185ffe-3de7-4b50-bcb0-cb23fb5abcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"
Sep 09 18:58:27.813512 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.813463    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n4h6\" (UniqueName: \"kubernetes.io/projected/77fa3cad-36b4-448d-b2ce-4a91e3a1d85a-kube-api-access-9n4h6\") pod \"payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps\" (UID: \"77fa3cad-36b4-448d-b2ce-4a91e3a1d85a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"
Sep 09 18:58:27.813698 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.813564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxlhx\" (UniqueName: \"kubernetes.io/projected/62185ffe-3de7-4b50-bcb0-cb23fb5abcd9-kube-api-access-bxlhx\") pod \"payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424\" (UID: \"62185ffe-3de7-4b50-bcb0-cb23fb5abcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"
Sep 09 18:58:27.813698 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.813597    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fa3cad-36b4-448d-b2ce-4a91e3a1d85a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps\" (UID: \"77fa3cad-36b4-448d-b2ce-4a91e3a1d85a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"
Sep 09 18:58:27.813698 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.813679    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62185ffe-3de7-4b50-bcb0-cb23fb5abcd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424\" (UID: \"62185ffe-3de7-4b50-bcb0-cb23fb5abcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"
Sep 09 18:58:27.814356 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.814337    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62185ffe-3de7-4b50-bcb0-cb23fb5abcd9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424\" (UID: \"62185ffe-3de7-4b50-bcb0-cb23fb5abcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"
Sep 09 18:58:27.822374 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.822345    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxlhx\" (UniqueName: \"kubernetes.io/projected/62185ffe-3de7-4b50-bcb0-cb23fb5abcd9-kube-api-access-bxlhx\") pod \"payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424\" (UID: \"62185ffe-3de7-4b50-bcb0-cb23fb5abcd9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"
Sep 09 18:58:27.914305 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.914267    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n4h6\" (UniqueName: \"kubernetes.io/projected/77fa3cad-36b4-448d-b2ce-4a91e3a1d85a-kube-api-access-9n4h6\") pod \"payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps\" (UID: \"77fa3cad-36b4-448d-b2ce-4a91e3a1d85a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"
Sep 09 18:58:27.914561 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.914328    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fa3cad-36b4-448d-b2ce-4a91e3a1d85a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps\" (UID: \"77fa3cad-36b4-448d-b2ce-4a91e3a1d85a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"
Sep 09 18:58:27.915066 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.915044    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fa3cad-36b4-448d-b2ce-4a91e3a1d85a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps\" (UID: \"77fa3cad-36b4-448d-b2ce-4a91e3a1d85a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"
Sep 09 18:58:27.923360 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.923298    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n4h6\" (UniqueName: \"kubernetes.io/projected/77fa3cad-36b4-448d-b2ce-4a91e3a1d85a-kube-api-access-9n4h6\") pod \"payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps\" (UID: \"77fa3cad-36b4-448d-b2ce-4a91e3a1d85a\") " pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"
Sep 09 18:58:27.985888 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:27.985833    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"
Sep 09 18:58:28.013254 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:28.013224    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"
Sep 09 18:58:28.264055 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:28.264029    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"]
Sep 09 18:58:28.266253 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:58:28.266228    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62185ffe_3de7_4b50_bcb0_cb23fb5abcd9.slice/crio-b7869f81daa3fbe194b1c2638c014066883c6e2e90ced72dd7dc3134c01ef188 WatchSource:0}: Error finding container b7869f81daa3fbe194b1c2638c014066883c6e2e90ced72dd7dc3134c01ef188: Status 404 returned error can't find the container with id b7869f81daa3fbe194b1c2638c014066883c6e2e90ced72dd7dc3134c01ef188
Sep 09 18:58:28.300882 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:28.300849    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"]
Sep 09 18:58:28.303862 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:58:28.303824    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fa3cad_36b4_448d_b2ce_4a91e3a1d85a.slice/crio-1e82978ef520cbf962c50ed143300512d6247a3b01af2b9bec2803f716a32618 WatchSource:0}: Error finding container 1e82978ef520cbf962c50ed143300512d6247a3b01af2b9bec2803f716a32618: Status 404 returned error can't find the container with id 1e82978ef520cbf962c50ed143300512d6247a3b01af2b9bec2803f716a32618
Sep 09 18:58:29.274658 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:29.274618    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424" event={"ID":"62185ffe-3de7-4b50-bcb0-cb23fb5abcd9","Type":"ContainerStarted","Data":"31a3c0f85aefb6bc10ee90a8e89b52373aa1bdff2487dedbec4bd6d04814191d"}
Sep 09 18:58:29.275181 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:29.274684    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424" event={"ID":"62185ffe-3de7-4b50-bcb0-cb23fb5abcd9","Type":"ContainerStarted","Data":"b7869f81daa3fbe194b1c2638c014066883c6e2e90ced72dd7dc3134c01ef188"}
Sep 09 18:58:29.275181 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:29.274785    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"
Sep 09 18:58:29.276376 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:29.276344    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps" event={"ID":"77fa3cad-36b4-448d-b2ce-4a91e3a1d85a","Type":"ContainerStarted","Data":"989e1ff638f839a0c4db64ef8b49baac8d1d907aad23356ba9583ff8b62de7d9"}
Sep 09 18:58:29.276537 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:29.276386    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps" event={"ID":"77fa3cad-36b4-448d-b2ce-4a91e3a1d85a","Type":"ContainerStarted","Data":"1e82978ef520cbf962c50ed143300512d6247a3b01af2b9bec2803f716a32618"}
Sep 09 18:58:29.276537 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:29.276416    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"
Sep 09 18:58:29.294471 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:29.294397    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424" podStartSLOduration=2.294374799 podStartE2EDuration="2.294374799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:58:29.291176049 +0000 UTC m=+1597.528545864" watchObservedRunningTime="2026-09-09 18:58:29.294374799 +0000 UTC m=+1597.531744616"
Sep 09 18:58:29.317063 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:29.317003    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps" podStartSLOduration=2.316984233 podStartE2EDuration="2.316984233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:58:29.315685811 +0000 UTC m=+1597.553055628" watchObservedRunningTime="2026-09-09 18:58:29.316984233 +0000 UTC m=+1597.554354044"
Sep 09 18:58:32.256118 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.256074    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd" podUID="5d84dd66-04b3-4254-8e6e-bbb464b1866c" containerName="main" containerID="cri-o://aa726ef4225b1d3bb3475dfcb395a3b6fc5f51188b8d3fc6e7d0e5fe77385a15" gracePeriod=45
Sep 09 18:58:32.524704 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.524672    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:58:32.665307 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.665266    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-dshm\") pod \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") "
Sep 09 18:58:32.665482 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.665340    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bbps\" (UniqueName: \"kubernetes.io/projected/5d84dd66-04b3-4254-8e6e-bbb464b1866c-kube-api-access-6bbps\") pod \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") "
Sep 09 18:58:32.665482 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.665404    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d84dd66-04b3-4254-8e6e-bbb464b1866c-tls-certs\") pod \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") "
Sep 09 18:58:32.665628 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.665574    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-home\") pod \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") "
Sep 09 18:58:32.665692 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.665643    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-tmp-dir\") pod \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") "
Sep 09 18:58:32.665692 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.665680    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-model-cache\") pod \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\" (UID: \"5d84dd66-04b3-4254-8e6e-bbb464b1866c\") "
Sep 09 18:58:32.665899 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.665829    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-home" (OuterVolumeSpecName: "home") pod "5d84dd66-04b3-4254-8e6e-bbb464b1866c" (UID: "5d84dd66-04b3-4254-8e6e-bbb464b1866c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:58:32.665899 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.665866    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d84dd66-04b3-4254-8e6e-bbb464b1866c" (UID: "5d84dd66-04b3-4254-8e6e-bbb464b1866c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:58:32.666106 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.665994    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d84dd66-04b3-4254-8e6e-bbb464b1866c" (UID: "5d84dd66-04b3-4254-8e6e-bbb464b1866c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:58:32.666167 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.666141    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:58:32.666167 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.666158    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:58:32.666271 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.666167    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:58:32.667651 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.667627    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d84dd66-04b3-4254-8e6e-bbb464b1866c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d84dd66-04b3-4254-8e6e-bbb464b1866c" (UID: "5d84dd66-04b3-4254-8e6e-bbb464b1866c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:58:32.667770 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.667675    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d84dd66-04b3-4254-8e6e-bbb464b1866c-kube-api-access-6bbps" (OuterVolumeSpecName: "kube-api-access-6bbps") pod "5d84dd66-04b3-4254-8e6e-bbb464b1866c" (UID: "5d84dd66-04b3-4254-8e6e-bbb464b1866c"). InnerVolumeSpecName "kube-api-access-6bbps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:58:32.668015 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.667998    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-dshm" (OuterVolumeSpecName: "dshm") pod "5d84dd66-04b3-4254-8e6e-bbb464b1866c" (UID: "5d84dd66-04b3-4254-8e6e-bbb464b1866c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:58:32.767610 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.767478    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d84dd66-04b3-4254-8e6e-bbb464b1866c-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:58:32.767610 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.767547    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bbps\" (UniqueName: \"kubernetes.io/projected/5d84dd66-04b3-4254-8e6e-bbb464b1866c-kube-api-access-6bbps\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:58:32.767610 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:32.767561    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d84dd66-04b3-4254-8e6e-bbb464b1866c-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:58:33.054286 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.054187    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh" podUID="945f5039-d593-4225-a050-fb4d9eb54c5e" containerName="main" containerID="cri-o://62f581ec92d61643e21617239a08bccba5b968e3c9b233f730388742f5b36e3b" gracePeriod=45
Sep 09 18:58:33.302433 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.302400    2575 generic.go:356] "Generic (PLEG): container finished" podID="5d84dd66-04b3-4254-8e6e-bbb464b1866c" containerID="aa726ef4225b1d3bb3475dfcb395a3b6fc5f51188b8d3fc6e7d0e5fe77385a15" exitCode=0
Sep 09 18:58:33.302881 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.302445    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"
Sep 09 18:58:33.302881 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.302487    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd" event={"ID":"5d84dd66-04b3-4254-8e6e-bbb464b1866c","Type":"ContainerDied","Data":"aa726ef4225b1d3bb3475dfcb395a3b6fc5f51188b8d3fc6e7d0e5fe77385a15"}
Sep 09 18:58:33.302881 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.302584    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd" event={"ID":"5d84dd66-04b3-4254-8e6e-bbb464b1866c","Type":"ContainerDied","Data":"db9a860aea5f0b331bcf1c2c5bf4a6efb4e81ec94cb38fc70d071233f17f3e02"}
Sep 09 18:58:33.302881 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.302607    2575 scope.go:117] "RemoveContainer" containerID="aa726ef4225b1d3bb3475dfcb395a3b6fc5f51188b8d3fc6e7d0e5fe77385a15"
Sep 09 18:58:33.304111 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.304089    2575 generic.go:356] "Generic (PLEG): container finished" podID="945f5039-d593-4225-a050-fb4d9eb54c5e" containerID="62f581ec92d61643e21617239a08bccba5b968e3c9b233f730388742f5b36e3b" exitCode=0
Sep 09 18:58:33.304206 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.304143    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh" event={"ID":"945f5039-d593-4225-a050-fb4d9eb54c5e","Type":"ContainerDied","Data":"62f581ec92d61643e21617239a08bccba5b968e3c9b233f730388742f5b36e3b"}
Sep 09 18:58:33.304206 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.304177    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh" event={"ID":"945f5039-d593-4225-a050-fb4d9eb54c5e","Type":"ContainerDied","Data":"bf0889e4b02641479ed3d5079492be67e4c3c37bde3c334dc0b8b3996e870d39"}
Sep 09 18:58:33.304206 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.304193    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf0889e4b02641479ed3d5079492be67e4c3c37bde3c334dc0b8b3996e870d39"
Sep 09 18:58:33.316071 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.316046    2575 scope.go:117] "RemoveContainer" containerID="aa726ef4225b1d3bb3475dfcb395a3b6fc5f51188b8d3fc6e7d0e5fe77385a15"
Sep 09 18:58:33.316407 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:58:33.316378    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa726ef4225b1d3bb3475dfcb395a3b6fc5f51188b8d3fc6e7d0e5fe77385a15\": container with ID starting with aa726ef4225b1d3bb3475dfcb395a3b6fc5f51188b8d3fc6e7d0e5fe77385a15 not found: ID does not exist" containerID="aa726ef4225b1d3bb3475dfcb395a3b6fc5f51188b8d3fc6e7d0e5fe77385a15"
Sep 09 18:58:33.316544 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.316419    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa726ef4225b1d3bb3475dfcb395a3b6fc5f51188b8d3fc6e7d0e5fe77385a15"} err="failed to get container status \"aa726ef4225b1d3bb3475dfcb395a3b6fc5f51188b8d3fc6e7d0e5fe77385a15\": rpc error: code = NotFound desc = could not find container \"aa726ef4225b1d3bb3475dfcb395a3b6fc5f51188b8d3fc6e7d0e5fe77385a15\": container with ID starting with aa726ef4225b1d3bb3475dfcb395a3b6fc5f51188b8d3fc6e7d0e5fe77385a15 not found: ID does not exist"
Sep 09 18:58:33.320598 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.320574    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:58:33.330866 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.330833    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"]
Sep 09 18:58:33.334713 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.334678    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/sub-test-model-282de3-kserve-56cb66f9-dkspd"]
Sep 09 18:58:33.474931 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.474892    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjb8g\" (UniqueName: \"kubernetes.io/projected/945f5039-d593-4225-a050-fb4d9eb54c5e-kube-api-access-rjb8g\") pod \"945f5039-d593-4225-a050-fb4d9eb54c5e\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") "
Sep 09 18:58:33.475196 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.474940    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-tmp-dir\") pod \"945f5039-d593-4225-a050-fb4d9eb54c5e\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") "
Sep 09 18:58:33.475196 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.474965    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/945f5039-d593-4225-a050-fb4d9eb54c5e-tls-certs\") pod \"945f5039-d593-4225-a050-fb4d9eb54c5e\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") "
Sep 09 18:58:33.475196 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.475030    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-dshm\") pod \"945f5039-d593-4225-a050-fb4d9eb54c5e\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") "
Sep 09 18:58:33.475196 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.475052    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-home\") pod \"945f5039-d593-4225-a050-fb4d9eb54c5e\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") "
Sep 09 18:58:33.475196 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.475159    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-model-cache\") pod \"945f5039-d593-4225-a050-fb4d9eb54c5e\" (UID: \"945f5039-d593-4225-a050-fb4d9eb54c5e\") "
Sep 09 18:58:33.475473 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.475268    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "945f5039-d593-4225-a050-fb4d9eb54c5e" (UID: "945f5039-d593-4225-a050-fb4d9eb54c5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:58:33.475473 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.475323    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-home" (OuterVolumeSpecName: "home") pod "945f5039-d593-4225-a050-fb4d9eb54c5e" (UID: "945f5039-d593-4225-a050-fb4d9eb54c5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:58:33.475473 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.475418    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "945f5039-d593-4225-a050-fb4d9eb54c5e" (UID: "945f5039-d593-4225-a050-fb4d9eb54c5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:58:33.475665 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.475590    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:58:33.475665 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.475608    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:58:33.475665 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.475622    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:58:33.477317 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.477288    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-dshm" (OuterVolumeSpecName: "dshm") pod "945f5039-d593-4225-a050-fb4d9eb54c5e" (UID: "945f5039-d593-4225-a050-fb4d9eb54c5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 18:58:33.477427 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.477319    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945f5039-d593-4225-a050-fb4d9eb54c5e-kube-api-access-rjb8g" (OuterVolumeSpecName: "kube-api-access-rjb8g") pod "945f5039-d593-4225-a050-fb4d9eb54c5e" (UID: "945f5039-d593-4225-a050-fb4d9eb54c5e"). InnerVolumeSpecName "kube-api-access-rjb8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:58:33.477427 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.477367    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945f5039-d593-4225-a050-fb4d9eb54c5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "945f5039-d593-4225-a050-fb4d9eb54c5e" (UID: "945f5039-d593-4225-a050-fb4d9eb54c5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 18:58:33.576936 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.576835    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjb8g\" (UniqueName: \"kubernetes.io/projected/945f5039-d593-4225-a050-fb4d9eb54c5e-kube-api-access-rjb8g\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:58:33.576936 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.576873    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/945f5039-d593-4225-a050-fb4d9eb54c5e-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:58:33.576936 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:33.576884    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/945f5039-d593-4225-a050-fb4d9eb54c5e-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:58:34.310691 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:34.310660    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"
Sep 09 18:58:34.321788 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:34.321757    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d84dd66-04b3-4254-8e6e-bbb464b1866c" path="/var/lib/kubelet/pods/5d84dd66-04b3-4254-8e6e-bbb464b1866c/volumes"
Sep 09 18:58:34.333355 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:34.333315    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"]
Sep 09 18:58:34.337890 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:34.337847    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/sub-test-model-0e012d-kserve-6f8f86c795-f7zmh"]
Sep 09 18:58:36.328478 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:36.328436    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="945f5039-d593-4225-a050-fb4d9eb54c5e" path="/var/lib/kubelet/pods/945f5039-d593-4225-a050-fb4d9eb54c5e/volumes"
Sep 09 18:58:40.284938 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:40.284903    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"
Sep 09 18:58:40.285525 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:40.284995    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"
Sep 09 18:58:53.205387 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.205345    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5"]
Sep 09 18:58:53.205876 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.205862    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d84dd66-04b3-4254-8e6e-bbb464b1866c" containerName="main"
Sep 09 18:58:53.205876 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.205877    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d84dd66-04b3-4254-8e6e-bbb464b1866c" containerName="main"
Sep 09 18:58:53.205957 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.205895    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945f5039-d593-4225-a050-fb4d9eb54c5e" containerName="main"
Sep 09 18:58:53.205957 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.205901    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="945f5039-d593-4225-a050-fb4d9eb54c5e" containerName="main"
Sep 09 18:58:53.206031 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.205983    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d84dd66-04b3-4254-8e6e-bbb464b1866c" containerName="main"
Sep 09 18:58:53.206031 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.205995    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="945f5039-d593-4225-a050-fb4d9eb54c5e" containerName="main"
Sep 09 18:58:53.209144 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.209124    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5"
Sep 09 18:58:53.211756 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.211722    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfzt2\""
Sep 09 18:58:53.211908 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.211721    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 18:58:53.216155 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.216119    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5"]
Sep 09 18:58:53.286615 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.286566    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7195870f-aa68-42d5-8ba5-896112693b20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5\" (UID: \"7195870f-aa68-42d5-8ba5-896112693b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5"
Sep 09 18:58:53.388281 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.388235    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7195870f-aa68-42d5-8ba5-896112693b20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5\" (UID: \"7195870f-aa68-42d5-8ba5-896112693b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5"
Sep 09 18:58:53.388957 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.388932    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7195870f-aa68-42d5-8ba5-896112693b20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5\" (UID: \"7195870f-aa68-42d5-8ba5-896112693b20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5"
Sep 09 18:58:53.521904 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.521798    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5"
Sep 09 18:58:53.799786 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.799751    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5"]
Sep 09 18:58:53.802161 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:58:53.802131    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7195870f_aa68_42d5_8ba5_896112693b20.slice/crio-24cdd8adde05ba2feb4e34c0fae61d7392ced3795b4f9331d2484c7fba08cf6d WatchSource:0}: Error finding container 24cdd8adde05ba2feb4e34c0fae61d7392ced3795b4f9331d2484c7fba08cf6d: Status 404 returned error can't find the container with id 24cdd8adde05ba2feb4e34c0fae61d7392ced3795b4f9331d2484c7fba08cf6d
Sep 09 18:58:53.804266 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:53.804245    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 18:58:54.430414 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:54.430371    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5" event={"ID":"7195870f-aa68-42d5-8ba5-896112693b20","Type":"ContainerStarted","Data":"24cdd8adde05ba2feb4e34c0fae61d7392ced3795b4f9331d2484c7fba08cf6d"}
Sep 09 18:58:57.452486 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:57.452377    2575 generic.go:356] "Generic (PLEG): container finished" podID="7195870f-aa68-42d5-8ba5-896112693b20" containerID="0717ce6127a0e56cb78a187908d8737f7d562efa84f465c45c35bcacb4f4bced" exitCode=0
Sep 09 18:58:57.452486 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:57.452449    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5" event={"ID":"7195870f-aa68-42d5-8ba5-896112693b20","Type":"ContainerDied","Data":"0717ce6127a0e56cb78a187908d8737f7d562efa84f465c45c35bcacb4f4bced"}
Sep 09 18:58:58.610282 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:58.610252    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5"
Sep 09 18:58:58.742589 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:58.742469    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7195870f-aa68-42d5-8ba5-896112693b20-maas-api-service-ca\") pod \"7195870f-aa68-42d5-8ba5-896112693b20\" (UID: \"7195870f-aa68-42d5-8ba5-896112693b20\") "
Sep 09 18:58:58.742836 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:58.742814    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7195870f-aa68-42d5-8ba5-896112693b20-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7195870f-aa68-42d5-8ba5-896112693b20" (UID: "7195870f-aa68-42d5-8ba5-896112693b20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:58:58.844204 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:58.844158    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7195870f-aa68-42d5-8ba5-896112693b20-maas-api-service-ca\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:58:59.466102 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:59.466073    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5"
Sep 09 18:58:59.466102 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:59.466096    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5" event={"ID":"7195870f-aa68-42d5-8ba5-896112693b20","Type":"ContainerDied","Data":"24cdd8adde05ba2feb4e34c0fae61d7392ced3795b4f9331d2484c7fba08cf6d"}
Sep 09 18:58:59.466334 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:58:59.466140    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24cdd8adde05ba2feb4e34c0fae61d7392ced3795b4f9331d2484c7fba08cf6d"
Sep 09 18:59:04.129516 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.129460    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"]
Sep 09 18:59:04.130020 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.130003    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7195870f-aa68-42d5-8ba5-896112693b20" containerName="revoke-keys"
Sep 09 18:59:04.130076 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.130022    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7195870f-aa68-42d5-8ba5-896112693b20" containerName="revoke-keys"
Sep 09 18:59:04.130112 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.130099    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7195870f-aa68-42d5-8ba5-896112693b20" containerName="revoke-keys"
Sep 09 18:59:04.134714 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.134685    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.138740 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.138707    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-282de3\"/\"rate-test-model-282de3-kserve-self-signed-certs\""
Sep 09 18:59:04.138740 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.138734    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-282de3\"/\"kube-root-ca.crt\""
Sep 09 18:59:04.139197 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.138764    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-282de3\"/\"openshift-service-ca.crt\""
Sep 09 18:59:04.139197 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.138787    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-282de3\"/\"default-dockercfg-bjp96\""
Sep 09 18:59:04.143086 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.143009    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"]
Sep 09 18:59:04.201406 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.201359    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-model-cache\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.201646 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.201429    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjqnc\" (UniqueName: \"kubernetes.io/projected/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-kube-api-access-wjqnc\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.201646 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.201469    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-dshm\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.201646 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.201564    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-tls-certs\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.201783 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.201646    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-home\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.201783 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.201680    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-tmp-dir\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.302860 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.302812    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-tmp-dir\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.302860 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.302864    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-model-cache\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.303133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.302905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjqnc\" (UniqueName: \"kubernetes.io/projected/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-kube-api-access-wjqnc\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.303133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.302951    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-dshm\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.303133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.302996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-tls-certs\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.303133 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.303104    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-home\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.303350 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.303301    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-tmp-dir\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.303350 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.303320    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-model-cache\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.303553 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.303530    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-home\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.305414 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.305394    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-dshm\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.305744 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.305721    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-tls-certs\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.312382 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.312342    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjqnc\" (UniqueName: \"kubernetes.io/projected/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-kube-api-access-wjqnc\") pod \"rate-test-model-282de3-kserve-59b55686cf-mzdmz\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") " pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.448031 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.447939    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:04.728986 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:04.728950    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"]
Sep 09 18:59:04.730805 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:59:04.730770    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded5841dc_3b89_4dbe_b7a7_ce91f00a75c5.slice/crio-4b7c82293ab6f8db797590f198c006c4fb760693f3863eeab545217af9c2aaac WatchSource:0}: Error finding container 4b7c82293ab6f8db797590f198c006c4fb760693f3863eeab545217af9c2aaac: Status 404 returned error can't find the container with id 4b7c82293ab6f8db797590f198c006c4fb760693f3863eeab545217af9c2aaac
Sep 09 18:59:05.501264 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:05.501228    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz" event={"ID":"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5","Type":"ContainerStarted","Data":"3d16b2d94f552bb8d27102fe639c65c88eff51b882cf29319d9b03caf62c66eb"}
Sep 09 18:59:05.501264 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:05.501270    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz" event={"ID":"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5","Type":"ContainerStarted","Data":"4b7c82293ab6f8db797590f198c006c4fb760693f3863eeab545217af9c2aaac"}
Sep 09 18:59:05.521075 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:05.521010    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz" podStartSLOduration=1.520990292 podStartE2EDuration="1.520990292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:59:05.51843441 +0000 UTC m=+1633.755804229" watchObservedRunningTime="2026-09-09 18:59:05.520990292 +0000 UTC m=+1633.758360108"
Sep 09 18:59:08.806062 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:08.806009    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"]
Sep 09 18:59:08.806633 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:08.806290    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762" podUID="7beb3fb8-51b9-46ba-be48-ed5f50533aef" containerName="payload-processing" containerID="cri-o://ef2dacb16b7f86dac2f5fee87dda3cad30cf56893eeebf1732282799fb1db1e8" gracePeriod=30
Sep 09 18:59:08.816450 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:08.816411    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"]
Sep 09 18:59:08.816910 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:08.816873    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2" podUID="83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7" containerName="payload-pre-processing" containerID="cri-o://49a2400627fe86de7720063d556a378e598d41f4cbc9017de509919c05cb7f37" gracePeriod=30
Sep 09 18:59:08.989829 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:08.989765    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5"]
Sep 09 18:59:08.992455 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:08.992421    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-79b1ac-xj9k5"]
Sep 09 18:59:09.116767 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.116735    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"
Sep 09 18:59:09.120883 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.120854    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"
Sep 09 18:59:09.252827 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.252786    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7beb3fb8-51b9-46ba-be48-ed5f50533aef-plugins-config-volume\") pod \"7beb3fb8-51b9-46ba-be48-ed5f50533aef\" (UID: \"7beb3fb8-51b9-46ba-be48-ed5f50533aef\") "
Sep 09 18:59:09.253036 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.252868    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52rwd\" (UniqueName: \"kubernetes.io/projected/7beb3fb8-51b9-46ba-be48-ed5f50533aef-kube-api-access-52rwd\") pod \"7beb3fb8-51b9-46ba-be48-ed5f50533aef\" (UID: \"7beb3fb8-51b9-46ba-be48-ed5f50533aef\") "
Sep 09 18:59:09.253036 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.252892    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7-plugins-config-volume\") pod \"83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7\" (UID: \"83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7\") "
Sep 09 18:59:09.253036 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.253015    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqllf\" (UniqueName: \"kubernetes.io/projected/83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7-kube-api-access-qqllf\") pod \"83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7\" (UID: \"83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7\") "
Sep 09 18:59:09.253267 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.253235    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7beb3fb8-51b9-46ba-be48-ed5f50533aef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7beb3fb8-51b9-46ba-be48-ed5f50533aef" (UID: "7beb3fb8-51b9-46ba-be48-ed5f50533aef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:59:09.253391 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.253364    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7" (UID: "83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:59:09.253460 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.253387    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7beb3fb8-51b9-46ba-be48-ed5f50533aef-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:59:09.255393 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.255358    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7-kube-api-access-qqllf" (OuterVolumeSpecName: "kube-api-access-qqllf") pod "83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7" (UID: "83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7"). InnerVolumeSpecName "kube-api-access-qqllf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:59:09.255586 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.255564    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7beb3fb8-51b9-46ba-be48-ed5f50533aef-kube-api-access-52rwd" (OuterVolumeSpecName: "kube-api-access-52rwd") pod "7beb3fb8-51b9-46ba-be48-ed5f50533aef" (UID: "7beb3fb8-51b9-46ba-be48-ed5f50533aef"). InnerVolumeSpecName "kube-api-access-52rwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:59:09.354566 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.354467    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqllf\" (UniqueName: \"kubernetes.io/projected/83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7-kube-api-access-qqllf\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:59:09.354566 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.354537    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52rwd\" (UniqueName: \"kubernetes.io/projected/7beb3fb8-51b9-46ba-be48-ed5f50533aef-kube-api-access-52rwd\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:59:09.354566 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.354548    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:59:09.524379 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.524334    2575 generic.go:356] "Generic (PLEG): container finished" podID="83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7" containerID="49a2400627fe86de7720063d556a378e598d41f4cbc9017de509919c05cb7f37" exitCode=0
Sep 09 18:59:09.524379 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.524376    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"
Sep 09 18:59:09.524655 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.524429    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2" event={"ID":"83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7","Type":"ContainerDied","Data":"49a2400627fe86de7720063d556a378e598d41f4cbc9017de509919c05cb7f37"}
Sep 09 18:59:09.524655 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.524490    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2" event={"ID":"83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7","Type":"ContainerDied","Data":"c9e036d7f4d40bbd98372223c7df1da67dc988c1e6cc875c2b8ddabf8f76764e"}
Sep 09 18:59:09.524655 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.524535    2575 scope.go:117] "RemoveContainer" containerID="49a2400627fe86de7720063d556a378e598d41f4cbc9017de509919c05cb7f37"
Sep 09 18:59:09.525927 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.525898    2575 generic.go:356] "Generic (PLEG): container finished" podID="7beb3fb8-51b9-46ba-be48-ed5f50533aef" containerID="ef2dacb16b7f86dac2f5fee87dda3cad30cf56893eeebf1732282799fb1db1e8" exitCode=0
Sep 09 18:59:09.526026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.525949    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"
Sep 09 18:59:09.526026 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.525977    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762" event={"ID":"7beb3fb8-51b9-46ba-be48-ed5f50533aef","Type":"ContainerDied","Data":"ef2dacb16b7f86dac2f5fee87dda3cad30cf56893eeebf1732282799fb1db1e8"}
Sep 09 18:59:09.526130 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.526036    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762" event={"ID":"7beb3fb8-51b9-46ba-be48-ed5f50533aef","Type":"ContainerDied","Data":"c011e5177f1b39553ed0e16347565ebd465ab7d60fd4fcf3c9d64c36e2a89047"}
Sep 09 18:59:09.534639 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.534606    2575 scope.go:117] "RemoveContainer" containerID="49a2400627fe86de7720063d556a378e598d41f4cbc9017de509919c05cb7f37"
Sep 09 18:59:09.535005 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:59:09.534982    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a2400627fe86de7720063d556a378e598d41f4cbc9017de509919c05cb7f37\": container with ID starting with 49a2400627fe86de7720063d556a378e598d41f4cbc9017de509919c05cb7f37 not found: ID does not exist" containerID="49a2400627fe86de7720063d556a378e598d41f4cbc9017de509919c05cb7f37"
Sep 09 18:59:09.535093 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.535013    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a2400627fe86de7720063d556a378e598d41f4cbc9017de509919c05cb7f37"} err="failed to get container status \"49a2400627fe86de7720063d556a378e598d41f4cbc9017de509919c05cb7f37\": rpc error: code = NotFound desc = could not find container \"49a2400627fe86de7720063d556a378e598d41f4cbc9017de509919c05cb7f37\": container with ID starting with 49a2400627fe86de7720063d556a378e598d41f4cbc9017de509919c05cb7f37 not found: ID does not exist"
Sep 09 18:59:09.535093 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.535033    2575 scope.go:117] "RemoveContainer" containerID="ef2dacb16b7f86dac2f5fee87dda3cad30cf56893eeebf1732282799fb1db1e8"
Sep 09 18:59:09.544314 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.544289    2575 scope.go:117] "RemoveContainer" containerID="ef2dacb16b7f86dac2f5fee87dda3cad30cf56893eeebf1732282799fb1db1e8"
Sep 09 18:59:09.544663 ip-10-0-134-63 kubenswrapper[2575]: E0909 18:59:09.544645    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef2dacb16b7f86dac2f5fee87dda3cad30cf56893eeebf1732282799fb1db1e8\": container with ID starting with ef2dacb16b7f86dac2f5fee87dda3cad30cf56893eeebf1732282799fb1db1e8 not found: ID does not exist" containerID="ef2dacb16b7f86dac2f5fee87dda3cad30cf56893eeebf1732282799fb1db1e8"
Sep 09 18:59:09.544728 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.544671    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef2dacb16b7f86dac2f5fee87dda3cad30cf56893eeebf1732282799fb1db1e8"} err="failed to get container status \"ef2dacb16b7f86dac2f5fee87dda3cad30cf56893eeebf1732282799fb1db1e8\": rpc error: code = NotFound desc = could not find container \"ef2dacb16b7f86dac2f5fee87dda3cad30cf56893eeebf1732282799fb1db1e8\": container with ID starting with ef2dacb16b7f86dac2f5fee87dda3cad30cf56893eeebf1732282799fb1db1e8 not found: ID does not exist"
Sep 09 18:59:09.548423 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.548390    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"]
Sep 09 18:59:09.554400 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.554357    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-79b1ac-7b69794f46-9b7z2"]
Sep 09 18:59:09.565833 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.565795    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"]
Sep 09 18:59:09.572234 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:09.572196    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-79b1ac-674cd9497c-bj762"]
Sep 09 18:59:10.321791 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:10.321751    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7195870f-aa68-42d5-8ba5-896112693b20" path="/var/lib/kubelet/pods/7195870f-aa68-42d5-8ba5-896112693b20/volumes"
Sep 09 18:59:10.322235 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:10.322215    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7beb3fb8-51b9-46ba-be48-ed5f50533aef" path="/var/lib/kubelet/pods/7beb3fb8-51b9-46ba-be48-ed5f50533aef/volumes"
Sep 09 18:59:10.322578 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:10.322563    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7" path="/var/lib/kubelet/pods/83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7/volumes"
Sep 09 18:59:14.449158 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:14.449123    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:14.449158 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:14.449167    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:14.462469 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:14.462437    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:14.566759 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:14.566727    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 18:59:15.878618 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:15.878575    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd"]
Sep 09 18:59:15.879044 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:15.879000    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7" containerName="payload-pre-processing"
Sep 09 18:59:15.879044 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:15.879013    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7" containerName="payload-pre-processing"
Sep 09 18:59:15.879147 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:15.879046    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7beb3fb8-51b9-46ba-be48-ed5f50533aef" containerName="payload-processing"
Sep 09 18:59:15.879147 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:15.879054    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7beb3fb8-51b9-46ba-be48-ed5f50533aef" containerName="payload-processing"
Sep 09 18:59:15.879147 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:15.879138    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7beb3fb8-51b9-46ba-be48-ed5f50533aef" containerName="payload-processing"
Sep 09 18:59:15.879258 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:15.879149    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="83db1a1e-e4f3-4ef0-ba09-e55adf1aefc7" containerName="payload-pre-processing"
Sep 09 18:59:15.883829 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:15.883800    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd"
Sep 09 18:59:15.886619 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:15.886581    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfzt2\""
Sep 09 18:59:15.886789 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:15.886603    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 18:59:15.890953 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:15.890923    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd"]
Sep 09 18:59:15.934179 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:15.934135    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abdf8c0-ef5c-40c7-b207-faac64f0dd4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd\" (UID: \"7abdf8c0-ef5c-40c7-b207-faac64f0dd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd"
Sep 09 18:59:16.035137 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:16.035091    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abdf8c0-ef5c-40c7-b207-faac64f0dd4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd\" (UID: \"7abdf8c0-ef5c-40c7-b207-faac64f0dd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd"
Sep 09 18:59:16.035806 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:16.035782    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abdf8c0-ef5c-40c7-b207-faac64f0dd4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd\" (UID: \"7abdf8c0-ef5c-40c7-b207-faac64f0dd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd"
Sep 09 18:59:16.196532 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:16.196380    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd"
Sep 09 18:59:16.475472 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:16.475441    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd"]
Sep 09 18:59:16.478011 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:59:16.477981    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abdf8c0_ef5c_40c7_b207_faac64f0dd4f.slice/crio-2ee6641f1b4ce53142d4a9eedbb7a50e23a71dbfd919741b2399debf00bb8df3 WatchSource:0}: Error finding container 2ee6641f1b4ce53142d4a9eedbb7a50e23a71dbfd919741b2399debf00bb8df3: Status 404 returned error can't find the container with id 2ee6641f1b4ce53142d4a9eedbb7a50e23a71dbfd919741b2399debf00bb8df3
Sep 09 18:59:16.567452 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:16.567413    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd" event={"ID":"7abdf8c0-ef5c-40c7-b207-faac64f0dd4f","Type":"ContainerStarted","Data":"2ee6641f1b4ce53142d4a9eedbb7a50e23a71dbfd919741b2399debf00bb8df3"}
Sep 09 18:59:17.574472 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:17.574378    2575 generic.go:356] "Generic (PLEG): container finished" podID="7abdf8c0-ef5c-40c7-b207-faac64f0dd4f" containerID="8ad14bd3bbcbec8bcec1a46f605b2d9e6b06ba94bd3d1dcf2a07f64c5eba7031" exitCode=0
Sep 09 18:59:17.574881 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:17.574470    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd" event={"ID":"7abdf8c0-ef5c-40c7-b207-faac64f0dd4f","Type":"ContainerDied","Data":"8ad14bd3bbcbec8bcec1a46f605b2d9e6b06ba94bd3d1dcf2a07f64c5eba7031"}
Sep 09 18:59:18.712786 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:18.712759    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd"
Sep 09 18:59:18.758923 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:18.758889    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abdf8c0-ef5c-40c7-b207-faac64f0dd4f-maas-api-service-ca\") pod \"7abdf8c0-ef5c-40c7-b207-faac64f0dd4f\" (UID: \"7abdf8c0-ef5c-40c7-b207-faac64f0dd4f\") "
Sep 09 18:59:18.759282 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:18.759258    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abdf8c0-ef5c-40c7-b207-faac64f0dd4f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7abdf8c0-ef5c-40c7-b207-faac64f0dd4f" (UID: "7abdf8c0-ef5c-40c7-b207-faac64f0dd4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:59:18.860593 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:18.860559    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abdf8c0-ef5c-40c7-b207-faac64f0dd4f-maas-api-service-ca\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:59:19.587740 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:19.587698    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd"
Sep 09 18:59:19.587930 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:19.587739    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd" event={"ID":"7abdf8c0-ef5c-40c7-b207-faac64f0dd4f","Type":"ContainerDied","Data":"2ee6641f1b4ce53142d4a9eedbb7a50e23a71dbfd919741b2399debf00bb8df3"}
Sep 09 18:59:19.587930 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:19.587792    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ee6641f1b4ce53142d4a9eedbb7a50e23a71dbfd919741b2399debf00bb8df3"
Sep 09 18:59:23.246090 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.246045    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"]
Sep 09 18:59:23.246546 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.246518    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abdf8c0-ef5c-40c7-b207-faac64f0dd4f" containerName="revoke-keys"
Sep 09 18:59:23.246546 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.246532    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abdf8c0-ef5c-40c7-b207-faac64f0dd4f" containerName="revoke-keys"
Sep 09 18:59:23.246648 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.246637    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abdf8c0-ef5c-40c7-b207-faac64f0dd4f" containerName="revoke-keys"
Sep 09 18:59:23.249861 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.249837    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.253407 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.253374    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e012d\"/\"openshift-service-ca.crt\""
Sep 09 18:59:23.253691 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.253454    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e012d\"/\"kube-root-ca.crt\""
Sep 09 18:59:23.253691 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.253462    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e012d\"/\"rate-test-model-0e012d-kserve-self-signed-certs\""
Sep 09 18:59:23.253691 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.253390    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e012d\"/\"default-dockercfg-6kjms\""
Sep 09 18:59:23.259471 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.259438    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"]
Sep 09 18:59:23.307907 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.307859    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76ccw\" (UniqueName: \"kubernetes.io/projected/56d31c34-42a8-40f1-bfde-69e3c5132be3-kube-api-access-76ccw\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.307907 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.307903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-home\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.308193 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.307943    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-model-cache\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.308193 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.307992    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-tmp-dir\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.308319 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.308205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d31c34-42a8-40f1-bfde-69e3c5132be3-tls-certs\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.308319 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.308244    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-dshm\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.409789 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.409731    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d31c34-42a8-40f1-bfde-69e3c5132be3-tls-certs\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.409789 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.409788    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-dshm\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.410083 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.409827    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76ccw\" (UniqueName: \"kubernetes.io/projected/56d31c34-42a8-40f1-bfde-69e3c5132be3-kube-api-access-76ccw\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.410083 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.409843    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-home\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.410083 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.409865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-model-cache\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.410083 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.409894    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-tmp-dir\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.410345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.410321    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-home\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.410345 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.410336    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-tmp-dir\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.410431 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.410392    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-model-cache\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.412336 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.412313    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-dshm\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.412524 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.412490    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d31c34-42a8-40f1-bfde-69e3c5132be3-tls-certs\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.417808 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.417777    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76ccw\" (UniqueName: \"kubernetes.io/projected/56d31c34-42a8-40f1-bfde-69e3c5132be3-kube-api-access-76ccw\") pod \"rate-test-model-0e012d-kserve-86d56f99f5-j4krl\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") " pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.562032 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.561935    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:23.842710 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:23.842681    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"]
Sep 09 18:59:23.845079 ip-10-0-134-63 kubenswrapper[2575]: W0909 18:59:23.845049    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56d31c34_42a8_40f1_bfde_69e3c5132be3.slice/crio-48661b47869291588f9a8429e710b09a58c984eb2b4c112a52fe7985a1e8796e WatchSource:0}: Error finding container 48661b47869291588f9a8429e710b09a58c984eb2b4c112a52fe7985a1e8796e: Status 404 returned error can't find the container with id 48661b47869291588f9a8429e710b09a58c984eb2b4c112a52fe7985a1e8796e
Sep 09 18:59:24.627016 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:24.626976    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl" event={"ID":"56d31c34-42a8-40f1-bfde-69e3c5132be3","Type":"ContainerStarted","Data":"a0d439da82ae11c8e140e045fcb683aeb0f5919f3c4b6a745dd53e5613456a95"}
Sep 09 18:59:24.627016 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:24.627022    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl" event={"ID":"56d31c34-42a8-40f1-bfde-69e3c5132be3","Type":"ContainerStarted","Data":"48661b47869291588f9a8429e710b09a58c984eb2b4c112a52fe7985a1e8796e"}
Sep 09 18:59:24.645463 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:24.645407    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl" podStartSLOduration=1.645387469 podStartE2EDuration="1.645387469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 18:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 18:59:24.645001625 +0000 UTC m=+1652.882371441" watchObservedRunningTime="2026-09-09 18:59:24.645387469 +0000 UTC m=+1652.882757284"
Sep 09 18:59:31.452485 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.452439    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"]
Sep 09 18:59:31.453077 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.452725    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps" podUID="77fa3cad-36b4-448d-b2ce-4a91e3a1d85a" containerName="payload-processing" containerID="cri-o://989e1ff638f839a0c4db64ef8b49baac8d1d907aad23356ba9583ff8b62de7d9" gracePeriod=30
Sep 09 18:59:31.457635 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.457602    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"]
Sep 09 18:59:31.457868 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.457831    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424" podUID="62185ffe-3de7-4b50-bcb0-cb23fb5abcd9" containerName="payload-pre-processing" containerID="cri-o://31a3c0f85aefb6bc10ee90a8e89b52373aa1bdff2487dedbec4bd6d04814191d" gracePeriod=30
Sep 09 18:59:31.630383 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.630342    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd"]
Sep 09 18:59:31.632486 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.632464    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fbc9c-ngvkd"]
Sep 09 18:59:31.669696 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.669657    2575 generic.go:356] "Generic (PLEG): container finished" podID="62185ffe-3de7-4b50-bcb0-cb23fb5abcd9" containerID="31a3c0f85aefb6bc10ee90a8e89b52373aa1bdff2487dedbec4bd6d04814191d" exitCode=0
Sep 09 18:59:31.670356 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.669740    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424" event={"ID":"62185ffe-3de7-4b50-bcb0-cb23fb5abcd9","Type":"ContainerDied","Data":"31a3c0f85aefb6bc10ee90a8e89b52373aa1bdff2487dedbec4bd6d04814191d"}
Sep 09 18:59:31.672180 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.672152    2575 generic.go:356] "Generic (PLEG): container finished" podID="77fa3cad-36b4-448d-b2ce-4a91e3a1d85a" containerID="989e1ff638f839a0c4db64ef8b49baac8d1d907aad23356ba9583ff8b62de7d9" exitCode=0
Sep 09 18:59:31.672307 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.672236    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps" event={"ID":"77fa3cad-36b4-448d-b2ce-4a91e3a1d85a","Type":"ContainerDied","Data":"989e1ff638f839a0c4db64ef8b49baac8d1d907aad23356ba9583ff8b62de7d9"}
Sep 09 18:59:31.747492 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.747463    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"
Sep 09 18:59:31.751113 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.751087    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"
Sep 09 18:59:31.802892 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.802859    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fa3cad-36b4-448d-b2ce-4a91e3a1d85a-plugins-config-volume\") pod \"77fa3cad-36b4-448d-b2ce-4a91e3a1d85a\" (UID: \"77fa3cad-36b4-448d-b2ce-4a91e3a1d85a\") "
Sep 09 18:59:31.802892 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.802901    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n4h6\" (UniqueName: \"kubernetes.io/projected/77fa3cad-36b4-448d-b2ce-4a91e3a1d85a-kube-api-access-9n4h6\") pod \"77fa3cad-36b4-448d-b2ce-4a91e3a1d85a\" (UID: \"77fa3cad-36b4-448d-b2ce-4a91e3a1d85a\") "
Sep 09 18:59:31.803270 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.803245    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fa3cad-36b4-448d-b2ce-4a91e3a1d85a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "77fa3cad-36b4-448d-b2ce-4a91e3a1d85a" (UID: "77fa3cad-36b4-448d-b2ce-4a91e3a1d85a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:59:31.805219 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.805192    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fa3cad-36b4-448d-b2ce-4a91e3a1d85a-kube-api-access-9n4h6" (OuterVolumeSpecName: "kube-api-access-9n4h6") pod "77fa3cad-36b4-448d-b2ce-4a91e3a1d85a" (UID: "77fa3cad-36b4-448d-b2ce-4a91e3a1d85a"). InnerVolumeSpecName "kube-api-access-9n4h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:59:31.904462 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.904425    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxlhx\" (UniqueName: \"kubernetes.io/projected/62185ffe-3de7-4b50-bcb0-cb23fb5abcd9-kube-api-access-bxlhx\") pod \"62185ffe-3de7-4b50-bcb0-cb23fb5abcd9\" (UID: \"62185ffe-3de7-4b50-bcb0-cb23fb5abcd9\") "
Sep 09 18:59:31.904684 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.904550    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62185ffe-3de7-4b50-bcb0-cb23fb5abcd9-plugins-config-volume\") pod \"62185ffe-3de7-4b50-bcb0-cb23fb5abcd9\" (UID: \"62185ffe-3de7-4b50-bcb0-cb23fb5abcd9\") "
Sep 09 18:59:31.904972 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.904941    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77fa3cad-36b4-448d-b2ce-4a91e3a1d85a-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:59:31.905042 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.904973    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n4h6\" (UniqueName: \"kubernetes.io/projected/77fa3cad-36b4-448d-b2ce-4a91e3a1d85a-kube-api-access-9n4h6\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:59:31.905042 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.904981    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62185ffe-3de7-4b50-bcb0-cb23fb5abcd9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62185ffe-3de7-4b50-bcb0-cb23fb5abcd9" (UID: "62185ffe-3de7-4b50-bcb0-cb23fb5abcd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 18:59:31.906714 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:31.906687    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62185ffe-3de7-4b50-bcb0-cb23fb5abcd9-kube-api-access-bxlhx" (OuterVolumeSpecName: "kube-api-access-bxlhx") pod "62185ffe-3de7-4b50-bcb0-cb23fb5abcd9" (UID: "62185ffe-3de7-4b50-bcb0-cb23fb5abcd9"). InnerVolumeSpecName "kube-api-access-bxlhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 18:59:32.006546 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:32.006412    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62185ffe-3de7-4b50-bcb0-cb23fb5abcd9-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:59:32.006546 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:32.006450    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxlhx\" (UniqueName: \"kubernetes.io/projected/62185ffe-3de7-4b50-bcb0-cb23fb5abcd9-kube-api-access-bxlhx\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 18:59:32.321688 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:32.321583    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7abdf8c0-ef5c-40c7-b207-faac64f0dd4f" path="/var/lib/kubelet/pods/7abdf8c0-ef5c-40c7-b207-faac64f0dd4f/volumes"
Sep 09 18:59:32.679779 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:32.679741    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"
Sep 09 18:59:32.679779 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:32.679758    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424" event={"ID":"62185ffe-3de7-4b50-bcb0-cb23fb5abcd9","Type":"ContainerDied","Data":"b7869f81daa3fbe194b1c2638c014066883c6e2e90ced72dd7dc3134c01ef188"}
Sep 09 18:59:32.680310 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:32.679825    2575 scope.go:117] "RemoveContainer" containerID="31a3c0f85aefb6bc10ee90a8e89b52373aa1bdff2487dedbec4bd6d04814191d"
Sep 09 18:59:32.681363 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:32.681340    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"
Sep 09 18:59:32.681479 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:32.681378    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps" event={"ID":"77fa3cad-36b4-448d-b2ce-4a91e3a1d85a","Type":"ContainerDied","Data":"1e82978ef520cbf962c50ed143300512d6247a3b01af2b9bec2803f716a32618"}
Sep 09 18:59:32.691163 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:32.691122    2575 scope.go:117] "RemoveContainer" containerID="989e1ff638f839a0c4db64ef8b49baac8d1d907aad23356ba9583ff8b62de7d9"
Sep 09 18:59:32.706491 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:32.706463    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"]
Sep 09 18:59:32.710812 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:32.710783    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8fbc9c-6d8b569b7f-j8rps"]
Sep 09 18:59:32.721811 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:32.721775    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"]
Sep 09 18:59:32.726194 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:32.726157    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8fbc9c-797f587466-mb424"]
Sep 09 18:59:33.563152 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:33.563115    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:33.563365 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:33.563170    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:33.575985 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:33.575953    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:33.701890 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:33.701850    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 18:59:34.321998 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:34.321963    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62185ffe-3de7-4b50-bcb0-cb23fb5abcd9" path="/var/lib/kubelet/pods/62185ffe-3de7-4b50-bcb0-cb23fb5abcd9/volumes"
Sep 09 18:59:34.322401 ip-10-0-134-63 kubenswrapper[2575]: I0909 18:59:34.322387    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77fa3cad-36b4-448d-b2ce-4a91e3a1d85a" path="/var/lib/kubelet/pods/77fa3cad-36b4-448d-b2ce-4a91e3a1d85a/volumes"
Sep 09 19:00:00.143890 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.143848    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e012d-29816340-wxzz9"]
Sep 09 19:00:00.144557 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.144535    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62185ffe-3de7-4b50-bcb0-cb23fb5abcd9" containerName="payload-pre-processing"
Sep 09 19:00:00.144557 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.144559    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="62185ffe-3de7-4b50-bcb0-cb23fb5abcd9" containerName="payload-pre-processing"
Sep 09 19:00:00.144706 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.144600    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77fa3cad-36b4-448d-b2ce-4a91e3a1d85a" containerName="payload-processing"
Sep 09 19:00:00.144706 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.144610    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="77fa3cad-36b4-448d-b2ce-4a91e3a1d85a" containerName="payload-processing"
Sep 09 19:00:00.144811 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.144723    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="62185ffe-3de7-4b50-bcb0-cb23fb5abcd9" containerName="payload-pre-processing"
Sep 09 19:00:00.144811 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.144735    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="77fa3cad-36b4-448d-b2ce-4a91e3a1d85a" containerName="payload-processing"
Sep 09 19:00:00.148157 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.148130    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-282de3-29816340-kkskr"]
Sep 09 19:00:00.148319 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.148284    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e012d-29816340-wxzz9"
Sep 09 19:00:00.156594 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.156557    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-282de3-29816340-kkskr"
Sep 09 19:00:00.157743 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.157715    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfzt2\""
Sep 09 19:00:00.163555 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.163525    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e012d-29816340-wxzz9"]
Sep 09 19:00:00.169945 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.169908    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-282de3-29816340-kkskr"]
Sep 09 19:00:00.170712 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.170686    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e012d-29816340-wxzz9"
Sep 09 19:00:00.182933 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.182904    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-282de3-29816340-kkskr"
Sep 09 19:00:00.467713 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.467685    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e012d-29816340-wxzz9"]
Sep 09 19:00:00.470279 ip-10-0-134-63 kubenswrapper[2575]: W0909 19:00:00.470249    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda04f77_87e7_43d7_8091_dfbc03cf0199.slice/crio-14c3e06f9923a873287ad5b390b57f5af12c81687ccadcf2f02f6b2ca489685b WatchSource:0}: Error finding container 14c3e06f9923a873287ad5b390b57f5af12c81687ccadcf2f02f6b2ca489685b: Status 404 returned error can't find the container with id 14c3e06f9923a873287ad5b390b57f5af12c81687ccadcf2f02f6b2ca489685b
Sep 09 19:00:00.480348 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.480325    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-282de3-29816340-kkskr"]
Sep 09 19:00:00.484647 ip-10-0-134-63 kubenswrapper[2575]: W0909 19:00:00.484620    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecfe888_50ff_4437_9f5d_4a7b73c1f748.slice/crio-abf6aa54c8458ef1604d8b670fcd8b17b8b46ff1dc96ab7a6b1822d3e18f907c WatchSource:0}: Error finding container abf6aa54c8458ef1604d8b670fcd8b17b8b46ff1dc96ab7a6b1822d3e18f907c: Status 404 returned error can't find the container with id abf6aa54c8458ef1604d8b670fcd8b17b8b46ff1dc96ab7a6b1822d3e18f907c
Sep 09 19:00:00.841678 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.841642    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e012d-29816340-wxzz9" event={"ID":"cda04f77-87e7-43d7-8091-dfbc03cf0199","Type":"ContainerStarted","Data":"1947209010c04572ed3adc65dfadbc340d904f1b7c7011f6c32a42dc3033d087"}
Sep 09 19:00:00.841887 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.841705    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e012d-29816340-wxzz9" event={"ID":"cda04f77-87e7-43d7-8091-dfbc03cf0199","Type":"ContainerStarted","Data":"14c3e06f9923a873287ad5b390b57f5af12c81687ccadcf2f02f6b2ca489685b"}
Sep 09 19:00:00.843322 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.843292    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-282de3-29816340-kkskr" event={"ID":"3ecfe888-50ff-4437-9f5d-4a7b73c1f748","Type":"ContainerStarted","Data":"15d56487e758919ae6da9b77255a7a2bdc269fee1bde12bba93f6b0fe739ecec"}
Sep 09 19:00:00.843472 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.843339    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-282de3-29816340-kkskr" event={"ID":"3ecfe888-50ff-4437-9f5d-4a7b73c1f748","Type":"ContainerStarted","Data":"abf6aa54c8458ef1604d8b670fcd8b17b8b46ff1dc96ab7a6b1822d3e18f907c"}
Sep 09 19:00:00.858232 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.858176    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e012d-29816340-wxzz9" podStartSLOduration=0.858158664 podStartE2EDuration="858.158664ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:00:00.856402238 +0000 UTC m=+1689.093772053" watchObservedRunningTime="2026-09-09 19:00:00.858158664 +0000 UTC m=+1689.095528520"
Sep 09 19:00:00.872614 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:00.872557    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-282de3-29816340-kkskr" podStartSLOduration=0.872537285 podStartE2EDuration="872.537285ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:00:00.870078176 +0000 UTC m=+1689.107447993" watchObservedRunningTime="2026-09-09 19:00:00.872537285 +0000 UTC m=+1689.109907104"
Sep 09 19:00:01.851697 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:01.851659    2575 generic.go:356] "Generic (PLEG): container finished" podID="3ecfe888-50ff-4437-9f5d-4a7b73c1f748" containerID="15d56487e758919ae6da9b77255a7a2bdc269fee1bde12bba93f6b0fe739ecec" exitCode=0
Sep 09 19:00:01.852199 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:01.851755    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-282de3-29816340-kkskr" event={"ID":"3ecfe888-50ff-4437-9f5d-4a7b73c1f748","Type":"ContainerDied","Data":"15d56487e758919ae6da9b77255a7a2bdc269fee1bde12bba93f6b0fe739ecec"}
Sep 09 19:00:01.853611 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:01.853575    2575 generic.go:356] "Generic (PLEG): container finished" podID="cda04f77-87e7-43d7-8091-dfbc03cf0199" containerID="1947209010c04572ed3adc65dfadbc340d904f1b7c7011f6c32a42dc3033d087" exitCode=0
Sep 09 19:00:01.853742 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:01.853650    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e012d-29816340-wxzz9" event={"ID":"cda04f77-87e7-43d7-8091-dfbc03cf0199","Type":"ContainerDied","Data":"1947209010c04572ed3adc65dfadbc340d904f1b7c7011f6c32a42dc3033d087"}
Sep 09 19:00:02.996939 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:02.996905    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e012d-29816340-wxzz9"
Sep 09 19:00:03.030434 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:03.030393    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-282de3-29816340-kkskr"
Sep 09 19:00:03.866562 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:03.866532    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e012d-29816340-wxzz9"
Sep 09 19:00:03.866562 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:03.866548    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e012d-29816340-wxzz9" event={"ID":"cda04f77-87e7-43d7-8091-dfbc03cf0199","Type":"ContainerDied","Data":"14c3e06f9923a873287ad5b390b57f5af12c81687ccadcf2f02f6b2ca489685b"}
Sep 09 19:00:03.866842 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:03.866601    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14c3e06f9923a873287ad5b390b57f5af12c81687ccadcf2f02f6b2ca489685b"
Sep 09 19:00:03.868107 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:03.868089    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-282de3-29816340-kkskr"
Sep 09 19:00:03.868217 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:03.868133    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-282de3-29816340-kkskr" event={"ID":"3ecfe888-50ff-4437-9f5d-4a7b73c1f748","Type":"ContainerDied","Data":"abf6aa54c8458ef1604d8b670fcd8b17b8b46ff1dc96ab7a6b1822d3e18f907c"}
Sep 09 19:00:03.868217 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:03.868170    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abf6aa54c8458ef1604d8b670fcd8b17b8b46ff1dc96ab7a6b1822d3e18f907c"
Sep 09 19:00:06.763171 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:06.763119    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"]
Sep 09 19:00:06.764225 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:06.763981    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94" podUID="5619f143-51d0-4897-94b9-a7e15c43a57f" containerName="payload-pre-processing" containerID="cri-o://f8e0e83ae2706aaad7ff36e5fb6b8972fbb9f5cb8d28c8bdf963802ae5ce0424" gracePeriod=30
Sep 09 19:00:06.814107 ip-10-0-134-63 kubenswrapper[2575]: E0909 19:00:06.814065    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5619f143_51d0_4897_94b9_a7e15c43a57f.slice/crio-conmon-f8e0e83ae2706aaad7ff36e5fb6b8972fbb9f5cb8d28c8bdf963802ae5ce0424.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5619f143_51d0_4897_94b9_a7e15c43a57f.slice/crio-f8e0e83ae2706aaad7ff36e5fb6b8972fbb9f5cb8d28c8bdf963802ae5ce0424.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 19:00:06.894607 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:06.894568    2575 generic.go:356] "Generic (PLEG): container finished" podID="5619f143-51d0-4897-94b9-a7e15c43a57f" containerID="f8e0e83ae2706aaad7ff36e5fb6b8972fbb9f5cb8d28c8bdf963802ae5ce0424" exitCode=0
Sep 09 19:00:06.894607 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:06.894600    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94" event={"ID":"5619f143-51d0-4897-94b9-a7e15c43a57f","Type":"ContainerDied","Data":"f8e0e83ae2706aaad7ff36e5fb6b8972fbb9f5cb8d28c8bdf963802ae5ce0424"}
Sep 09 19:00:07.029933 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:07.029854    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"
Sep 09 19:00:07.179251 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:07.179211    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5619f143-51d0-4897-94b9-a7e15c43a57f-plugins-config-volume\") pod \"5619f143-51d0-4897-94b9-a7e15c43a57f\" (UID: \"5619f143-51d0-4897-94b9-a7e15c43a57f\") "
Sep 09 19:00:07.179423 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:07.179402    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l57f8\" (UniqueName: \"kubernetes.io/projected/5619f143-51d0-4897-94b9-a7e15c43a57f-kube-api-access-l57f8\") pod \"5619f143-51d0-4897-94b9-a7e15c43a57f\" (UID: \"5619f143-51d0-4897-94b9-a7e15c43a57f\") "
Sep 09 19:00:07.179689 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:07.179664    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5619f143-51d0-4897-94b9-a7e15c43a57f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5619f143-51d0-4897-94b9-a7e15c43a57f" (UID: "5619f143-51d0-4897-94b9-a7e15c43a57f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:00:07.179794 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:07.179779    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5619f143-51d0-4897-94b9-a7e15c43a57f-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:00:07.181830 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:07.181797    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5619f143-51d0-4897-94b9-a7e15c43a57f-kube-api-access-l57f8" (OuterVolumeSpecName: "kube-api-access-l57f8") pod "5619f143-51d0-4897-94b9-a7e15c43a57f" (UID: "5619f143-51d0-4897-94b9-a7e15c43a57f"). InnerVolumeSpecName "kube-api-access-l57f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:00:07.280684 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:07.280584    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l57f8\" (UniqueName: \"kubernetes.io/projected/5619f143-51d0-4897-94b9-a7e15c43a57f-kube-api-access-l57f8\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:00:07.902841 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:07.902422    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94" event={"ID":"5619f143-51d0-4897-94b9-a7e15c43a57f","Type":"ContainerDied","Data":"a009376dbcfc4f74fee05dd6874dfd54bc05e02f6ceb2144fa816a1886aebef8"}
Sep 09 19:00:07.902841 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:07.902538    2575 scope.go:117] "RemoveContainer" containerID="f8e0e83ae2706aaad7ff36e5fb6b8972fbb9f5cb8d28c8bdf963802ae5ce0424"
Sep 09 19:00:07.902841 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:07.902597    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"
Sep 09 19:00:07.930675 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:07.930636    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"]
Sep 09 19:00:07.934320 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:07.934282    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c09a03-745675b898-zvw94"]
Sep 09 19:00:08.323194 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:08.323110    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5619f143-51d0-4897-94b9-a7e15c43a57f" path="/var/lib/kubelet/pods/5619f143-51d0-4897-94b9-a7e15c43a57f/volumes"
Sep 09 19:00:08.638071 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:08.638033    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"]
Sep 09 19:00:09.648845 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:09.648799    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"]
Sep 09 19:00:23.658695 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.658625    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz" podUID="ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5" containerName="main" containerID="cri-o://3d16b2d94f552bb8d27102fe639c65c88eff51b882cf29319d9b03caf62c66eb" gracePeriod=45
Sep 09 19:00:23.912482 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.912405    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 19:00:23.953691 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.953646    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-tmp-dir\") pod \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") "
Sep 09 19:00:23.953905 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.953762    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-tls-certs\") pod \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") "
Sep 09 19:00:23.953905 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.953812    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-home\") pod \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") "
Sep 09 19:00:23.953905 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.953882    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-dshm\") pod \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") "
Sep 09 19:00:23.954075 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.953922    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-model-cache\") pod \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") "
Sep 09 19:00:23.954075 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.953969    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5" (UID: "ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:00:23.954075 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.953982    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjqnc\" (UniqueName: \"kubernetes.io/projected/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-kube-api-access-wjqnc\") pod \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\" (UID: \"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5\") "
Sep 09 19:00:23.954217 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.954090    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-home" (OuterVolumeSpecName: "home") pod "ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5" (UID: "ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:00:23.954468 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.954439    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:00:23.954468 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.954469    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:00:23.954696 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.954536    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5" (UID: "ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:00:23.956292 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.956267    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-kube-api-access-wjqnc" (OuterVolumeSpecName: "kube-api-access-wjqnc") pod "ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5" (UID: "ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5"). InnerVolumeSpecName "kube-api-access-wjqnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:00:23.956575 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.956551    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5" (UID: "ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:00:23.956575 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.956565    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-dshm" (OuterVolumeSpecName: "dshm") pod "ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5" (UID: "ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:00:23.994645 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.994606    2575 generic.go:356] "Generic (PLEG): container finished" podID="ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5" containerID="3d16b2d94f552bb8d27102fe639c65c88eff51b882cf29319d9b03caf62c66eb" exitCode=0
Sep 09 19:00:23.994835 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.994660    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"
Sep 09 19:00:23.994835 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.994701    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz" event={"ID":"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5","Type":"ContainerDied","Data":"3d16b2d94f552bb8d27102fe639c65c88eff51b882cf29319d9b03caf62c66eb"}
Sep 09 19:00:23.994835 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.994752    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz" event={"ID":"ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5","Type":"ContainerDied","Data":"4b7c82293ab6f8db797590f198c006c4fb760693f3863eeab545217af9c2aaac"}
Sep 09 19:00:23.994835 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:23.994770    2575 scope.go:117] "RemoveContainer" containerID="3d16b2d94f552bb8d27102fe639c65c88eff51b882cf29319d9b03caf62c66eb"
Sep 09 19:00:24.004713 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.004686    2575 scope.go:117] "RemoveContainer" containerID="3d16b2d94f552bb8d27102fe639c65c88eff51b882cf29319d9b03caf62c66eb"
Sep 09 19:00:24.005055 ip-10-0-134-63 kubenswrapper[2575]: E0909 19:00:24.005032    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d16b2d94f552bb8d27102fe639c65c88eff51b882cf29319d9b03caf62c66eb\": container with ID starting with 3d16b2d94f552bb8d27102fe639c65c88eff51b882cf29319d9b03caf62c66eb not found: ID does not exist" containerID="3d16b2d94f552bb8d27102fe639c65c88eff51b882cf29319d9b03caf62c66eb"
Sep 09 19:00:24.005128 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.005070    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d16b2d94f552bb8d27102fe639c65c88eff51b882cf29319d9b03caf62c66eb"} err="failed to get container status \"3d16b2d94f552bb8d27102fe639c65c88eff51b882cf29319d9b03caf62c66eb\": rpc error: code = NotFound desc = could not find container \"3d16b2d94f552bb8d27102fe639c65c88eff51b882cf29319d9b03caf62c66eb\": container with ID starting with 3d16b2d94f552bb8d27102fe639c65c88eff51b882cf29319d9b03caf62c66eb not found: ID does not exist"
Sep 09 19:00:24.017565 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.017525    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"]
Sep 09 19:00:24.019703 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.019679    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-282de3/rate-test-model-282de3-kserve-59b55686cf-mzdmz"]
Sep 09 19:00:24.055058 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.055020    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:00:24.055058 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.055051    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:00:24.055058 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.055062    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjqnc\" (UniqueName: \"kubernetes.io/projected/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-kube-api-access-wjqnc\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:00:24.055258 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.055072    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:00:24.321183 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.321091    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5" path="/var/lib/kubelet/pods/ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5/volumes"
Sep 09 19:00:24.670043 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.670003    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl" podUID="56d31c34-42a8-40f1-bfde-69e3c5132be3" containerName="main" containerID="cri-o://a0d439da82ae11c8e140e045fcb683aeb0f5919f3c4b6a745dd53e5613456a95" gracePeriod=45
Sep 09 19:00:24.924893 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.924811    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 19:00:24.965102 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.965055    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-tmp-dir\") pod \"56d31c34-42a8-40f1-bfde-69e3c5132be3\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") "
Sep 09 19:00:24.965310 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.965122    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76ccw\" (UniqueName: \"kubernetes.io/projected/56d31c34-42a8-40f1-bfde-69e3c5132be3-kube-api-access-76ccw\") pod \"56d31c34-42a8-40f1-bfde-69e3c5132be3\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") "
Sep 09 19:00:24.965310 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.965162    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-model-cache\") pod \"56d31c34-42a8-40f1-bfde-69e3c5132be3\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") "
Sep 09 19:00:24.965310 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.965202    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-dshm\") pod \"56d31c34-42a8-40f1-bfde-69e3c5132be3\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") "
Sep 09 19:00:24.965310 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.965250    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-home\") pod \"56d31c34-42a8-40f1-bfde-69e3c5132be3\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") "
Sep 09 19:00:24.965310 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.965271    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d31c34-42a8-40f1-bfde-69e3c5132be3-tls-certs\") pod \"56d31c34-42a8-40f1-bfde-69e3c5132be3\" (UID: \"56d31c34-42a8-40f1-bfde-69e3c5132be3\") "
Sep 09 19:00:24.965617 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.965392    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56d31c34-42a8-40f1-bfde-69e3c5132be3" (UID: "56d31c34-42a8-40f1-bfde-69e3c5132be3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:00:24.965617 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.965467    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-model-cache" (OuterVolumeSpecName: "model-cache") pod "56d31c34-42a8-40f1-bfde-69e3c5132be3" (UID: "56d31c34-42a8-40f1-bfde-69e3c5132be3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:00:24.965930 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.965702    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-home" (OuterVolumeSpecName: "home") pod "56d31c34-42a8-40f1-bfde-69e3c5132be3" (UID: "56d31c34-42a8-40f1-bfde-69e3c5132be3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:00:24.965930 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.965760    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:00:24.965930 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.965783    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:00:24.967917 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.967888    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-dshm" (OuterVolumeSpecName: "dshm") pod "56d31c34-42a8-40f1-bfde-69e3c5132be3" (UID: "56d31c34-42a8-40f1-bfde-69e3c5132be3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:00:24.967917 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.967903    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d31c34-42a8-40f1-bfde-69e3c5132be3-kube-api-access-76ccw" (OuterVolumeSpecName: "kube-api-access-76ccw") pod "56d31c34-42a8-40f1-bfde-69e3c5132be3" (UID: "56d31c34-42a8-40f1-bfde-69e3c5132be3"). InnerVolumeSpecName "kube-api-access-76ccw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:00:24.968103 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:24.968016    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d31c34-42a8-40f1-bfde-69e3c5132be3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56d31c34-42a8-40f1-bfde-69e3c5132be3" (UID: "56d31c34-42a8-40f1-bfde-69e3c5132be3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:00:25.002943 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:25.002904    2575 generic.go:356] "Generic (PLEG): container finished" podID="56d31c34-42a8-40f1-bfde-69e3c5132be3" containerID="a0d439da82ae11c8e140e045fcb683aeb0f5919f3c4b6a745dd53e5613456a95" exitCode=0
Sep 09 19:00:25.003149 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:25.002967    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"
Sep 09 19:00:25.003149 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:25.002983    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl" event={"ID":"56d31c34-42a8-40f1-bfde-69e3c5132be3","Type":"ContainerDied","Data":"a0d439da82ae11c8e140e045fcb683aeb0f5919f3c4b6a745dd53e5613456a95"}
Sep 09 19:00:25.003149 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:25.003016    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl" event={"ID":"56d31c34-42a8-40f1-bfde-69e3c5132be3","Type":"ContainerDied","Data":"48661b47869291588f9a8429e710b09a58c984eb2b4c112a52fe7985a1e8796e"}
Sep 09 19:00:25.003149 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:25.003040    2575 scope.go:117] "RemoveContainer" containerID="a0d439da82ae11c8e140e045fcb683aeb0f5919f3c4b6a745dd53e5613456a95"
Sep 09 19:00:25.013638 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:25.013600    2575 scope.go:117] "RemoveContainer" containerID="a0d439da82ae11c8e140e045fcb683aeb0f5919f3c4b6a745dd53e5613456a95"
Sep 09 19:00:25.013947 ip-10-0-134-63 kubenswrapper[2575]: E0909 19:00:25.013929    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d439da82ae11c8e140e045fcb683aeb0f5919f3c4b6a745dd53e5613456a95\": container with ID starting with a0d439da82ae11c8e140e045fcb683aeb0f5919f3c4b6a745dd53e5613456a95 not found: ID does not exist" containerID="a0d439da82ae11c8e140e045fcb683aeb0f5919f3c4b6a745dd53e5613456a95"
Sep 09 19:00:25.013994 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:25.013958    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d439da82ae11c8e140e045fcb683aeb0f5919f3c4b6a745dd53e5613456a95"} err="failed to get container status \"a0d439da82ae11c8e140e045fcb683aeb0f5919f3c4b6a745dd53e5613456a95\": rpc error: code = NotFound desc = could not find container \"a0d439da82ae11c8e140e045fcb683aeb0f5919f3c4b6a745dd53e5613456a95\": container with ID starting with a0d439da82ae11c8e140e045fcb683aeb0f5919f3c4b6a745dd53e5613456a95 not found: ID does not exist"
Sep 09 19:00:25.025933 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:25.025898    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"]
Sep 09 19:00:25.029898 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:25.029866    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e012d/rate-test-model-0e012d-kserve-86d56f99f5-j4krl"]
Sep 09 19:00:25.067289 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:25.067254    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76ccw\" (UniqueName: \"kubernetes.io/projected/56d31c34-42a8-40f1-bfde-69e3c5132be3-kube-api-access-76ccw\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:00:25.067289 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:25.067285    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:00:25.067289 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:25.067296    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56d31c34-42a8-40f1-bfde-69e3c5132be3-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:00:25.067566 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:25.067304    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d31c34-42a8-40f1-bfde-69e3c5132be3-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:00:26.322639 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:26.322596    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56d31c34-42a8-40f1-bfde-69e3c5132be3" path="/var/lib/kubelet/pods/56d31c34-42a8-40f1-bfde-69e3c5132be3/volumes"
Sep 09 19:00:44.375041 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375003    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"]
Sep 09 19:00:44.375553 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375414    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ecfe888-50ff-4437-9f5d-4a7b73c1f748" containerName="cleanup"
Sep 09 19:00:44.375553 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375426    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ecfe888-50ff-4437-9f5d-4a7b73c1f748" containerName="cleanup"
Sep 09 19:00:44.375553 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375440    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5619f143-51d0-4897-94b9-a7e15c43a57f" containerName="payload-pre-processing"
Sep 09 19:00:44.375553 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375447    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5619f143-51d0-4897-94b9-a7e15c43a57f" containerName="payload-pre-processing"
Sep 09 19:00:44.375553 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375457    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda04f77-87e7-43d7-8091-dfbc03cf0199" containerName="cleanup"
Sep 09 19:00:44.375553 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375463    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda04f77-87e7-43d7-8091-dfbc03cf0199" containerName="cleanup"
Sep 09 19:00:44.375553 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375469    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5" containerName="main"
Sep 09 19:00:44.375553 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375474    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5" containerName="main"
Sep 09 19:00:44.375553 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375485    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d31c34-42a8-40f1-bfde-69e3c5132be3" containerName="main"
Sep 09 19:00:44.375553 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375490    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d31c34-42a8-40f1-bfde-69e3c5132be3" containerName="main"
Sep 09 19:00:44.375896 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375568    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed5841dc-3b89-4dbe-b7a7-ce91f00a75c5" containerName="main"
Sep 09 19:00:44.375896 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375575    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d31c34-42a8-40f1-bfde-69e3c5132be3" containerName="main"
Sep 09 19:00:44.375896 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375582    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ecfe888-50ff-4437-9f5d-4a7b73c1f748" containerName="cleanup"
Sep 09 19:00:44.375896 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375589    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5619f143-51d0-4897-94b9-a7e15c43a57f" containerName="payload-pre-processing"
Sep 09 19:00:44.375896 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.375596    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cda04f77-87e7-43d7-8091-dfbc03cf0199" containerName="cleanup"
Sep 09 19:00:44.380274 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.380251    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"
Sep 09 19:00:44.383147 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.383105    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-03128a\""
Sep 09 19:00:44.383481 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.383460    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-03128a-dockercfg-5kv8h\""
Sep 09 19:00:44.389822 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.389793    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"]
Sep 09 19:00:44.424664 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.424624    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"]
Sep 09 19:00:44.428221 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.428200    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"
Sep 09 19:00:44.440566 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.440535    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"]
Sep 09 19:00:44.459891 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.459851    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2ec485-7c66-4dde-8cdf-59daf91a6c86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6\" (UID: \"ca2ec485-7c66-4dde-8cdf-59daf91a6c86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"
Sep 09 19:00:44.460289 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.460256    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dc6x\" (UniqueName: \"kubernetes.io/projected/ca2ec485-7c66-4dde-8cdf-59daf91a6c86-kube-api-access-4dc6x\") pod \"payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6\" (UID: \"ca2ec485-7c66-4dde-8cdf-59daf91a6c86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"
Sep 09 19:00:44.561955 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.561907    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edec93d2-7c1a-4f4a-97bf-15477f71133f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-03128a-789f459677-frqst\" (UID: \"edec93d2-7c1a-4f4a-97bf-15477f71133f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"
Sep 09 19:00:44.562185 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.561979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjzxx\" (UniqueName: \"kubernetes.io/projected/edec93d2-7c1a-4f4a-97bf-15477f71133f-kube-api-access-wjzxx\") pod \"payload-processing-e2e-ipp-b-03128a-789f459677-frqst\" (UID: \"edec93d2-7c1a-4f4a-97bf-15477f71133f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"
Sep 09 19:00:44.562185 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.562036    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dc6x\" (UniqueName: \"kubernetes.io/projected/ca2ec485-7c66-4dde-8cdf-59daf91a6c86-kube-api-access-4dc6x\") pod \"payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6\" (UID: \"ca2ec485-7c66-4dde-8cdf-59daf91a6c86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"
Sep 09 19:00:44.562185 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.562129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2ec485-7c66-4dde-8cdf-59daf91a6c86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6\" (UID: \"ca2ec485-7c66-4dde-8cdf-59daf91a6c86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"
Sep 09 19:00:44.562814 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.562792    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2ec485-7c66-4dde-8cdf-59daf91a6c86-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6\" (UID: \"ca2ec485-7c66-4dde-8cdf-59daf91a6c86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"
Sep 09 19:00:44.571794 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.571758    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dc6x\" (UniqueName: \"kubernetes.io/projected/ca2ec485-7c66-4dde-8cdf-59daf91a6c86-kube-api-access-4dc6x\") pod \"payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6\" (UID: \"ca2ec485-7c66-4dde-8cdf-59daf91a6c86\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"
Sep 09 19:00:44.663309 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.663194    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edec93d2-7c1a-4f4a-97bf-15477f71133f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-03128a-789f459677-frqst\" (UID: \"edec93d2-7c1a-4f4a-97bf-15477f71133f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"
Sep 09 19:00:44.663309 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.663281    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjzxx\" (UniqueName: \"kubernetes.io/projected/edec93d2-7c1a-4f4a-97bf-15477f71133f-kube-api-access-wjzxx\") pod \"payload-processing-e2e-ipp-b-03128a-789f459677-frqst\" (UID: \"edec93d2-7c1a-4f4a-97bf-15477f71133f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"
Sep 09 19:00:44.664012 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.663991    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edec93d2-7c1a-4f4a-97bf-15477f71133f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-03128a-789f459677-frqst\" (UID: \"edec93d2-7c1a-4f4a-97bf-15477f71133f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"
Sep 09 19:00:44.673265 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.673231    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjzxx\" (UniqueName: \"kubernetes.io/projected/edec93d2-7c1a-4f4a-97bf-15477f71133f-kube-api-access-wjzxx\") pod \"payload-processing-e2e-ipp-b-03128a-789f459677-frqst\" (UID: \"edec93d2-7c1a-4f4a-97bf-15477f71133f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"
Sep 09 19:00:44.692727 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.692689    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"
Sep 09 19:00:44.740314 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.740253    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"
Sep 09 19:00:44.972164 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:44.972135    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"]
Sep 09 19:00:44.974513 ip-10-0-134-63 kubenswrapper[2575]: W0909 19:00:44.974472    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2ec485_7c66_4dde_8cdf_59daf91a6c86.slice/crio-8276e270ada137c3ac2898f8fb595c2e462e883133cce577d9f747270514d2aa WatchSource:0}: Error finding container 8276e270ada137c3ac2898f8fb595c2e462e883133cce577d9f747270514d2aa: Status 404 returned error can't find the container with id 8276e270ada137c3ac2898f8fb595c2e462e883133cce577d9f747270514d2aa
Sep 09 19:00:45.032796 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:45.032763    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"]
Sep 09 19:00:45.034778 ip-10-0-134-63 kubenswrapper[2575]: W0909 19:00:45.034749    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedec93d2_7c1a_4f4a_97bf_15477f71133f.slice/crio-e204ea6efb4d4938158a50d4591f729c43fdfeff25060e0fc112cbc445bdc2f8 WatchSource:0}: Error finding container e204ea6efb4d4938158a50d4591f729c43fdfeff25060e0fc112cbc445bdc2f8: Status 404 returned error can't find the container with id e204ea6efb4d4938158a50d4591f729c43fdfeff25060e0fc112cbc445bdc2f8
Sep 09 19:00:45.118526 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:45.118462    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6" event={"ID":"ca2ec485-7c66-4dde-8cdf-59daf91a6c86","Type":"ContainerStarted","Data":"8aef7b95b3c4d9f060e2daebc39323509fe6e5f28e404f7546a74ed29f6eef4f"}
Sep 09 19:00:45.118725 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:45.118549    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6" event={"ID":"ca2ec485-7c66-4dde-8cdf-59daf91a6c86","Type":"ContainerStarted","Data":"8276e270ada137c3ac2898f8fb595c2e462e883133cce577d9f747270514d2aa"}
Sep 09 19:00:45.118725 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:45.118580    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"
Sep 09 19:00:45.120416 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:45.120387    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst" event={"ID":"edec93d2-7c1a-4f4a-97bf-15477f71133f","Type":"ContainerStarted","Data":"4007fcc7a6dd9b07a6a97a4fa4ce87e0fce6a2bd6b89aa177cf7ecdfa3607f9a"}
Sep 09 19:00:45.120583 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:45.120454    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"
Sep 09 19:00:45.120583 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:45.120477    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst" event={"ID":"edec93d2-7c1a-4f4a-97bf-15477f71133f","Type":"ContainerStarted","Data":"e204ea6efb4d4938158a50d4591f729c43fdfeff25060e0fc112cbc445bdc2f8"}
Sep 09 19:00:45.136058 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:45.136001    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6" podStartSLOduration=1.135981519 podStartE2EDuration="1.135981519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:00:45.13446263 +0000 UTC m=+1733.371832446" watchObservedRunningTime="2026-09-09 19:00:45.135981519 +0000 UTC m=+1733.373351337"
Sep 09 19:00:45.152427 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:45.152351    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst" podStartSLOduration=1.152325427 podStartE2EDuration="1.152325427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:00:45.15185051 +0000 UTC m=+1733.389220336" watchObservedRunningTime="2026-09-09 19:00:45.152325427 +0000 UTC m=+1733.389695258"
Sep 09 19:00:56.128833 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:56.128798    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"
Sep 09 19:00:56.129298 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:00:56.128871    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"
Sep 09 19:01:12.474176 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:12.474137    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-b4lhf_69804e5e-c502-42ae-aee8-d4e2f7ea2262/payload-processing/0.log"
Sep 09 19:01:12.597122 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:12.597074    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-03128a-789f459677-frqst_edec93d2-7c1a-4f4a-97bf-15477f71133f/payload-processing/0.log"
Sep 09 19:01:19.145598 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.145554    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"]
Sep 09 19:01:19.149724 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.149696    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.153556 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.153531    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a63380\"/\"ipp-route-a63380-kserve-self-signed-certs\""
Sep 09 19:01:19.153710 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.153531    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a63380\"/\"kube-root-ca.crt\""
Sep 09 19:01:19.153710 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.153531    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-a63380\"/\"default-dockercfg-j6kxm\""
Sep 09 19:01:19.153710 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.153573    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-a63380\"/\"openshift-service-ca.crt\""
Sep 09 19:01:19.158634 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.158523    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"]
Sep 09 19:01:19.309029 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.308987    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-tmp-dir\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.309029 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.309033    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-home\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.309256 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.309071    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-model-cache\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.309256 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.309109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nc42\" (UniqueName: \"kubernetes.io/projected/bc1491f5-7e37-4071-b157-6c88cc14c13d-kube-api-access-5nc42\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.309256 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.309213    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-dshm\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.309256 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.309249    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1491f5-7e37-4071-b157-6c88cc14c13d-tls-certs\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.410351 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.410243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-tmp-dir\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.410351 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.410291    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-home\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.410351 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.410321    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-model-cache\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.410351 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.410347    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nc42\" (UniqueName: \"kubernetes.io/projected/bc1491f5-7e37-4071-b157-6c88cc14c13d-kube-api-access-5nc42\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.410780 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.410413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-dshm\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.410780 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.410431    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1491f5-7e37-4071-b157-6c88cc14c13d-tls-certs\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.410780 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.410715    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-tmp-dir\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.410909 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.410794    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-home\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.410909 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.410828    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-model-cache\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.412864 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.412843    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-dshm\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.413167 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.413149    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1491f5-7e37-4071-b157-6c88cc14c13d-tls-certs\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.418726 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.418694    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nc42\" (UniqueName: \"kubernetes.io/projected/bc1491f5-7e37-4071-b157-6c88cc14c13d-kube-api-access-5nc42\") pod \"ipp-route-a63380-kserve-58998997f5-rnnm2\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") " pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.461769 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.461729    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:19.742657 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:19.742625    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"]
Sep 09 19:01:19.744741 ip-10-0-134-63 kubenswrapper[2575]: W0909 19:01:19.744712    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc1491f5_7e37_4071_b157_6c88cc14c13d.slice/crio-1be7df39630f4c96ac287dd6656b56b74aad3e83e7feebcf402f986d838c0991 WatchSource:0}: Error finding container 1be7df39630f4c96ac287dd6656b56b74aad3e83e7feebcf402f986d838c0991: Status 404 returned error can't find the container with id 1be7df39630f4c96ac287dd6656b56b74aad3e83e7feebcf402f986d838c0991
Sep 09 19:01:20.323595 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:20.323551    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2" event={"ID":"bc1491f5-7e37-4071-b157-6c88cc14c13d","Type":"ContainerStarted","Data":"94795a3d7281697d74829fffe3da5235dccc594e0831a69e801cd05155463f06"}
Sep 09 19:01:20.324107 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:20.323607    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2" event={"ID":"bc1491f5-7e37-4071-b157-6c88cc14c13d","Type":"ContainerStarted","Data":"1be7df39630f4c96ac287dd6656b56b74aad3e83e7feebcf402f986d838c0991"}
Sep 09 19:01:20.342649 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:20.342587    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2" podStartSLOduration=1.342567496 podStartE2EDuration="1.342567496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:01:20.340435827 +0000 UTC m=+1768.577805655" watchObservedRunningTime="2026-09-09 19:01:20.342567496 +0000 UTC m=+1768.579937310"
Sep 09 19:01:29.462649 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:29.462592    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:29.463212 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:29.462668    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:29.475438 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:29.475406    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:30.388760 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:30.388726    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:01:48.435648 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:48.435550    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-b4lhf_69804e5e-c502-42ae-aee8-d4e2f7ea2262/payload-processing/0.log"
Sep 09 19:01:48.553907 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:48.553878    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-03128a-789f459677-frqst_edec93d2-7c1a-4f4a-97bf-15477f71133f/payload-processing/0.log"
Sep 09 19:01:52.383588 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:52.383556    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 19:01:52.388345 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:52.388317    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 19:01:53.841059 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:53.841021    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"]
Sep 09 19:01:53.844730 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:53.844709    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"
Sep 09 19:01:53.848061 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:53.848033    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-5ae86e-dockercfg-zhjxm\""
Sep 09 19:01:53.848258 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:53.848239    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-5ae86e\""
Sep 09 19:01:53.861466 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:53.861432    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"]
Sep 09 19:01:53.972222 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:53.972179    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dd7l\" (UniqueName: \"kubernetes.io/projected/39b6741a-ac17-4ac5-bc77-ab1f361a2428-kube-api-access-7dd7l\") pod \"payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf\" (UID: \"39b6741a-ac17-4ac5-bc77-ab1f361a2428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"
Sep 09 19:01:53.972222 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:53.972230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b6741a-ac17-4ac5-bc77-ab1f361a2428-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf\" (UID: \"39b6741a-ac17-4ac5-bc77-ab1f361a2428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"
Sep 09 19:01:54.073675 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:54.073630    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b6741a-ac17-4ac5-bc77-ab1f361a2428-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf\" (UID: \"39b6741a-ac17-4ac5-bc77-ab1f361a2428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"
Sep 09 19:01:54.073858 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:54.073821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dd7l\" (UniqueName: \"kubernetes.io/projected/39b6741a-ac17-4ac5-bc77-ab1f361a2428-kube-api-access-7dd7l\") pod \"payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf\" (UID: \"39b6741a-ac17-4ac5-bc77-ab1f361a2428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"
Sep 09 19:01:54.074558 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:54.074539    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b6741a-ac17-4ac5-bc77-ab1f361a2428-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf\" (UID: \"39b6741a-ac17-4ac5-bc77-ab1f361a2428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"
Sep 09 19:01:54.082414 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:54.082375    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dd7l\" (UniqueName: \"kubernetes.io/projected/39b6741a-ac17-4ac5-bc77-ab1f361a2428-kube-api-access-7dd7l\") pod \"payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf\" (UID: \"39b6741a-ac17-4ac5-bc77-ab1f361a2428\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"
Sep 09 19:01:54.155847 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:54.155802    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"
Sep 09 19:01:54.435231 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:54.435199    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"]
Sep 09 19:01:54.436768 ip-10-0-134-63 kubenswrapper[2575]: W0909 19:01:54.436733    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b6741a_ac17_4ac5_bc77_ab1f361a2428.slice/crio-d951ece9ee8a8ed1bed8485a0bd5a6fb2985b1ce978d4543858475053b137eb7 WatchSource:0}: Error finding container d951ece9ee8a8ed1bed8485a0bd5a6fb2985b1ce978d4543858475053b137eb7: Status 404 returned error can't find the container with id d951ece9ee8a8ed1bed8485a0bd5a6fb2985b1ce978d4543858475053b137eb7
Sep 09 19:01:54.515915 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:54.515886    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf" event={"ID":"39b6741a-ac17-4ac5-bc77-ab1f361a2428","Type":"ContainerStarted","Data":"d951ece9ee8a8ed1bed8485a0bd5a6fb2985b1ce978d4543858475053b137eb7"}
Sep 09 19:01:55.522701 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:55.522660    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf" event={"ID":"39b6741a-ac17-4ac5-bc77-ab1f361a2428","Type":"ContainerStarted","Data":"f21c940f2d2322df825a8a0cc34bbc8adb4c31a6f8eeb2d6caf9db114508a54b"}
Sep 09 19:01:55.523125 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:55.522799    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"
Sep 09 19:01:55.541764 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:01:55.541706    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf" podStartSLOduration=2.541688787 podStartE2EDuration="2.541688787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:01:55.538854406 +0000 UTC m=+1803.776224227" watchObservedRunningTime="2026-09-09 19:01:55.541688787 +0000 UTC m=+1803.779058602"
Sep 09 19:02:06.531492 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:06.531460    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"
Sep 09 19:02:17.700367 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:17.700325    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f"]
Sep 09 19:02:17.704194 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:17.704172    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f"
Sep 09 19:02:17.706960 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:17.706929    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfzt2\""
Sep 09 19:02:17.707091 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:17.706929    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 19:02:17.711597 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:17.711568    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f"]
Sep 09 19:02:17.806029 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:17.805983    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16eb54a-ed8f-4fcc-883e-7d63b90a4b47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f\" (UID: \"f16eb54a-ed8f-4fcc-883e-7d63b90a4b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f"
Sep 09 19:02:17.906758 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:17.906711    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16eb54a-ed8f-4fcc-883e-7d63b90a4b47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f\" (UID: \"f16eb54a-ed8f-4fcc-883e-7d63b90a4b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f"
Sep 09 19:02:17.907412 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:17.907388    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16eb54a-ed8f-4fcc-883e-7d63b90a4b47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f\" (UID: \"f16eb54a-ed8f-4fcc-883e-7d63b90a4b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f"
Sep 09 19:02:18.017145 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:18.017037    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f"
Sep 09 19:02:18.293336 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:18.293308    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f"]
Sep 09 19:02:18.295586 ip-10-0-134-63 kubenswrapper[2575]: W0909 19:02:18.295552    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16eb54a_ed8f_4fcc_883e_7d63b90a4b47.slice/crio-83322240895b49d784e3635b09c832b8d077618a77efcb3ea5286f7607847147 WatchSource:0}: Error finding container 83322240895b49d784e3635b09c832b8d077618a77efcb3ea5286f7607847147: Status 404 returned error can't find the container with id 83322240895b49d784e3635b09c832b8d077618a77efcb3ea5286f7607847147
Sep 09 19:02:18.656620 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:18.656574    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f" event={"ID":"f16eb54a-ed8f-4fcc-883e-7d63b90a4b47","Type":"ContainerStarted","Data":"cfdb836dea9be36795a6fde0413d0ac479938fd472346058f7cc9bd05778eadf"}
Sep 09 19:02:18.656806 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:18.656648    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f" event={"ID":"f16eb54a-ed8f-4fcc-883e-7d63b90a4b47","Type":"ContainerStarted","Data":"83322240895b49d784e3635b09c832b8d077618a77efcb3ea5286f7607847147"}
Sep 09 19:02:18.673076 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:18.673020    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f" podStartSLOduration=1.67300155 podStartE2EDuration="1.67300155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:02:18.672127014 +0000 UTC m=+1826.909496828" watchObservedRunningTime="2026-09-09 19:02:18.67300155 +0000 UTC m=+1826.910371365"
Sep 09 19:02:19.663754 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:19.663709    2575 generic.go:356] "Generic (PLEG): container finished" podID="f16eb54a-ed8f-4fcc-883e-7d63b90a4b47" containerID="cfdb836dea9be36795a6fde0413d0ac479938fd472346058f7cc9bd05778eadf" exitCode=0
Sep 09 19:02:19.664175 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:19.663792    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f" event={"ID":"f16eb54a-ed8f-4fcc-883e-7d63b90a4b47","Type":"ContainerDied","Data":"cfdb836dea9be36795a6fde0413d0ac479938fd472346058f7cc9bd05778eadf"}
Sep 09 19:02:20.812164 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:20.812135    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f"
Sep 09 19:02:20.938582 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:20.938473    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16eb54a-ed8f-4fcc-883e-7d63b90a4b47-maas-api-service-ca\") pod \"f16eb54a-ed8f-4fcc-883e-7d63b90a4b47\" (UID: \"f16eb54a-ed8f-4fcc-883e-7d63b90a4b47\") "
Sep 09 19:02:20.938875 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:20.938849    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16eb54a-ed8f-4fcc-883e-7d63b90a4b47-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f16eb54a-ed8f-4fcc-883e-7d63b90a4b47" (UID: "f16eb54a-ed8f-4fcc-883e-7d63b90a4b47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:02:21.040033 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:21.039987    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16eb54a-ed8f-4fcc-883e-7d63b90a4b47-maas-api-service-ca\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:02:21.675802 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:21.675768    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f"
Sep 09 19:02:21.676056 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:21.675807    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f" event={"ID":"f16eb54a-ed8f-4fcc-883e-7d63b90a4b47","Type":"ContainerDied","Data":"83322240895b49d784e3635b09c832b8d077618a77efcb3ea5286f7607847147"}
Sep 09 19:02:21.676056 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:21.675856    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83322240895b49d784e3635b09c832b8d077618a77efcb3ea5286f7607847147"
Sep 09 19:02:23.762521 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:23.762462    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"]
Sep 09 19:02:23.763027 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:23.762761    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf" podUID="39b6741a-ac17-4ac5-bc77-ab1f361a2428" containerName="payload-pre-processing" containerID="cri-o://f21c940f2d2322df825a8a0cc34bbc8adb4c31a6f8eeb2d6caf9db114508a54b" gracePeriod=30
Sep 09 19:02:23.926694 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:23.926649    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f"]
Sep 09 19:02:23.929096 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:23.929062    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5ae86e-sr24f"]
Sep 09 19:02:24.029871 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.029801    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"
Sep 09 19:02:24.173942 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.173895    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b6741a-ac17-4ac5-bc77-ab1f361a2428-plugins-config-volume\") pod \"39b6741a-ac17-4ac5-bc77-ab1f361a2428\" (UID: \"39b6741a-ac17-4ac5-bc77-ab1f361a2428\") "
Sep 09 19:02:24.174216 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.173996    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dd7l\" (UniqueName: \"kubernetes.io/projected/39b6741a-ac17-4ac5-bc77-ab1f361a2428-kube-api-access-7dd7l\") pod \"39b6741a-ac17-4ac5-bc77-ab1f361a2428\" (UID: \"39b6741a-ac17-4ac5-bc77-ab1f361a2428\") "
Sep 09 19:02:24.174458 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.174393    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b6741a-ac17-4ac5-bc77-ab1f361a2428-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39b6741a-ac17-4ac5-bc77-ab1f361a2428" (UID: "39b6741a-ac17-4ac5-bc77-ab1f361a2428"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:02:24.176316 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.176291    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b6741a-ac17-4ac5-bc77-ab1f361a2428-kube-api-access-7dd7l" (OuterVolumeSpecName: "kube-api-access-7dd7l") pod "39b6741a-ac17-4ac5-bc77-ab1f361a2428" (UID: "39b6741a-ac17-4ac5-bc77-ab1f361a2428"). InnerVolumeSpecName "kube-api-access-7dd7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:02:24.275144 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.275099    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b6741a-ac17-4ac5-bc77-ab1f361a2428-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:02:24.275144 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.275134    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dd7l\" (UniqueName: \"kubernetes.io/projected/39b6741a-ac17-4ac5-bc77-ab1f361a2428-kube-api-access-7dd7l\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:02:24.321555 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.321444    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f16eb54a-ed8f-4fcc-883e-7d63b90a4b47" path="/var/lib/kubelet/pods/f16eb54a-ed8f-4fcc-883e-7d63b90a4b47/volumes"
Sep 09 19:02:24.695578 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.695542    2575 generic.go:356] "Generic (PLEG): container finished" podID="39b6741a-ac17-4ac5-bc77-ab1f361a2428" containerID="f21c940f2d2322df825a8a0cc34bbc8adb4c31a6f8eeb2d6caf9db114508a54b" exitCode=0
Sep 09 19:02:24.695804 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.695600    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"
Sep 09 19:02:24.695804 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.695631    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf" event={"ID":"39b6741a-ac17-4ac5-bc77-ab1f361a2428","Type":"ContainerDied","Data":"f21c940f2d2322df825a8a0cc34bbc8adb4c31a6f8eeb2d6caf9db114508a54b"}
Sep 09 19:02:24.695804 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.695688    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf" event={"ID":"39b6741a-ac17-4ac5-bc77-ab1f361a2428","Type":"ContainerDied","Data":"d951ece9ee8a8ed1bed8485a0bd5a6fb2985b1ce978d4543858475053b137eb7"}
Sep 09 19:02:24.695804 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.695717    2575 scope.go:117] "RemoveContainer" containerID="f21c940f2d2322df825a8a0cc34bbc8adb4c31a6f8eeb2d6caf9db114508a54b"
Sep 09 19:02:24.706057 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.706037    2575 scope.go:117] "RemoveContainer" containerID="f21c940f2d2322df825a8a0cc34bbc8adb4c31a6f8eeb2d6caf9db114508a54b"
Sep 09 19:02:24.706388 ip-10-0-134-63 kubenswrapper[2575]: E0909 19:02:24.706372    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21c940f2d2322df825a8a0cc34bbc8adb4c31a6f8eeb2d6caf9db114508a54b\": container with ID starting with f21c940f2d2322df825a8a0cc34bbc8adb4c31a6f8eeb2d6caf9db114508a54b not found: ID does not exist" containerID="f21c940f2d2322df825a8a0cc34bbc8adb4c31a6f8eeb2d6caf9db114508a54b"
Sep 09 19:02:24.706430 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.706398    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21c940f2d2322df825a8a0cc34bbc8adb4c31a6f8eeb2d6caf9db114508a54b"} err="failed to get container status \"f21c940f2d2322df825a8a0cc34bbc8adb4c31a6f8eeb2d6caf9db114508a54b\": rpc error: code = NotFound desc = could not find container \"f21c940f2d2322df825a8a0cc34bbc8adb4c31a6f8eeb2d6caf9db114508a54b\": container with ID starting with f21c940f2d2322df825a8a0cc34bbc8adb4c31a6f8eeb2d6caf9db114508a54b not found: ID does not exist"
Sep 09 19:02:24.714681 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.714645    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"]
Sep 09 19:02:24.719010 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:24.718979    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ae86e-87b48d98-8w2cf"]
Sep 09 19:02:26.321566 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:26.321534    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39b6741a-ac17-4ac5-bc77-ab1f361a2428" path="/var/lib/kubelet/pods/39b6741a-ac17-4ac5-bc77-ab1f361a2428/volumes"
Sep 09 19:02:34.047626 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.047590    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr"]
Sep 09 19:02:34.048108 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.048092    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b6741a-ac17-4ac5-bc77-ab1f361a2428" containerName="payload-pre-processing"
Sep 09 19:02:34.048162 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.048111    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b6741a-ac17-4ac5-bc77-ab1f361a2428" containerName="payload-pre-processing"
Sep 09 19:02:34.048162 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.048126    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f16eb54a-ed8f-4fcc-883e-7d63b90a4b47" containerName="revoke-keys"
Sep 09 19:02:34.048162 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.048134    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f16eb54a-ed8f-4fcc-883e-7d63b90a4b47" containerName="revoke-keys"
Sep 09 19:02:34.048279 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.048209    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b6741a-ac17-4ac5-bc77-ab1f361a2428" containerName="payload-pre-processing"
Sep 09 19:02:34.048279 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.048219    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f16eb54a-ed8f-4fcc-883e-7d63b90a4b47" containerName="revoke-keys"
Sep 09 19:02:34.051177 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.051161    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr"
Sep 09 19:02:34.053693 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.053671    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 19:02:34.053824 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.053785    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfzt2\""
Sep 09 19:02:34.059116 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.058676    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr"]
Sep 09 19:02:34.178432 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.178384    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb501848-afe9-4546-9b7a-f5358aaa9611-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr\" (UID: \"eb501848-afe9-4546-9b7a-f5358aaa9611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr"
Sep 09 19:02:34.279327 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.279284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb501848-afe9-4546-9b7a-f5358aaa9611-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr\" (UID: \"eb501848-afe9-4546-9b7a-f5358aaa9611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr"
Sep 09 19:02:34.280002 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.279981    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb501848-afe9-4546-9b7a-f5358aaa9611-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr\" (UID: \"eb501848-afe9-4546-9b7a-f5358aaa9611\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr"
Sep 09 19:02:34.363079 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.363047    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr"
Sep 09 19:02:34.638091 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.638052    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr"]
Sep 09 19:02:34.640076 ip-10-0-134-63 kubenswrapper[2575]: W0909 19:02:34.640038    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb501848_afe9_4546_9b7a_f5358aaa9611.slice/crio-b96a875496b0607793c43573b586c145a6b03bb0d759c66e9919d35317719d73 WatchSource:0}: Error finding container b96a875496b0607793c43573b586c145a6b03bb0d759c66e9919d35317719d73: Status 404 returned error can't find the container with id b96a875496b0607793c43573b586c145a6b03bb0d759c66e9919d35317719d73
Sep 09 19:02:34.758887 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:34.758840    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr" event={"ID":"eb501848-afe9-4546-9b7a-f5358aaa9611","Type":"ContainerStarted","Data":"b96a875496b0607793c43573b586c145a6b03bb0d759c66e9919d35317719d73"}
Sep 09 19:02:35.767255 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:35.767161    2575 generic.go:356] "Generic (PLEG): container finished" podID="eb501848-afe9-4546-9b7a-f5358aaa9611" containerID="ffc66a0a3fe55863d76dc7abb6a6b6f55438338ced9c43d78459e05768732955" exitCode=0
Sep 09 19:02:35.767685 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:35.767244    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr" event={"ID":"eb501848-afe9-4546-9b7a-f5358aaa9611","Type":"ContainerDied","Data":"ffc66a0a3fe55863d76dc7abb6a6b6f55438338ced9c43d78459e05768732955"}
Sep 09 19:02:36.918807 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:36.918782    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr"
Sep 09 19:02:37.003156 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:37.003116    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb501848-afe9-4546-9b7a-f5358aaa9611-maas-api-service-ca\") pod \"eb501848-afe9-4546-9b7a-f5358aaa9611\" (UID: \"eb501848-afe9-4546-9b7a-f5358aaa9611\") "
Sep 09 19:02:37.003458 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:37.003435    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb501848-afe9-4546-9b7a-f5358aaa9611-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb501848-afe9-4546-9b7a-f5358aaa9611" (UID: "eb501848-afe9-4546-9b7a-f5358aaa9611"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:02:37.104776 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:37.104690    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb501848-afe9-4546-9b7a-f5358aaa9611-maas-api-service-ca\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:02:37.780586 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:37.780547    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr" event={"ID":"eb501848-afe9-4546-9b7a-f5358aaa9611","Type":"ContainerDied","Data":"b96a875496b0607793c43573b586c145a6b03bb0d759c66e9919d35317719d73"}
Sep 09 19:02:37.780752 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:37.780606    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b96a875496b0607793c43573b586c145a6b03bb0d759c66e9919d35317719d73"
Sep 09 19:02:37.780752 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:37.780565    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr"
Sep 09 19:02:49.872929 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:49.872888    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr"]
Sep 09 19:02:49.875587 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:49.875556    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a63380-l4gmr"]
Sep 09 19:02:50.321752 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:50.321659    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb501848-afe9-4546-9b7a-f5358aaa9611" path="/var/lib/kubelet/pods/eb501848-afe9-4546-9b7a-f5358aaa9611/volumes"
Sep 09 19:02:55.227808 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:02:55.227770    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"]
Sep 09 19:03:10.248326 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.248274    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2" podUID="bc1491f5-7e37-4071-b157-6c88cc14c13d" containerName="main" containerID="cri-o://94795a3d7281697d74829fffe3da5235dccc594e0831a69e801cd05155463f06" gracePeriod=45
Sep 09 19:03:10.377485 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.377440    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2" podUID="bc1491f5-7e37-4071-b157-6c88cc14c13d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": dial tcp 10.134.0.83:8000: connect: connection refused"
Sep 09 19:03:10.509581 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.509514    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:03:10.555659 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.555621    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-tmp-dir\") pod \"bc1491f5-7e37-4071-b157-6c88cc14c13d\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") "
Sep 09 19:03:10.555860 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.555699    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-model-cache\") pod \"bc1491f5-7e37-4071-b157-6c88cc14c13d\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") "
Sep 09 19:03:10.555860 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.555796    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-dshm\") pod \"bc1491f5-7e37-4071-b157-6c88cc14c13d\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") "
Sep 09 19:03:10.555986 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.555870    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1491f5-7e37-4071-b157-6c88cc14c13d-tls-certs\") pod \"bc1491f5-7e37-4071-b157-6c88cc14c13d\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") "
Sep 09 19:03:10.555986 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.555922    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nc42\" (UniqueName: \"kubernetes.io/projected/bc1491f5-7e37-4071-b157-6c88cc14c13d-kube-api-access-5nc42\") pod \"bc1491f5-7e37-4071-b157-6c88cc14c13d\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") "
Sep 09 19:03:10.555986 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.555951    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-home\") pod \"bc1491f5-7e37-4071-b157-6c88cc14c13d\" (UID: \"bc1491f5-7e37-4071-b157-6c88cc14c13d\") "
Sep 09 19:03:10.555986 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.555953    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc1491f5-7e37-4071-b157-6c88cc14c13d" (UID: "bc1491f5-7e37-4071-b157-6c88cc14c13d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:03:10.556355 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.556260    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:03:10.556355 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.556341    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc1491f5-7e37-4071-b157-6c88cc14c13d" (UID: "bc1491f5-7e37-4071-b157-6c88cc14c13d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:03:10.556556 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.556537    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-home" (OuterVolumeSpecName: "home") pod "bc1491f5-7e37-4071-b157-6c88cc14c13d" (UID: "bc1491f5-7e37-4071-b157-6c88cc14c13d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:03:10.558486 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.558451    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-dshm" (OuterVolumeSpecName: "dshm") pod "bc1491f5-7e37-4071-b157-6c88cc14c13d" (UID: "bc1491f5-7e37-4071-b157-6c88cc14c13d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:03:10.558486 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.558459    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1491f5-7e37-4071-b157-6c88cc14c13d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc1491f5-7e37-4071-b157-6c88cc14c13d" (UID: "bc1491f5-7e37-4071-b157-6c88cc14c13d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:03:10.558696 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.558585    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1491f5-7e37-4071-b157-6c88cc14c13d-kube-api-access-5nc42" (OuterVolumeSpecName: "kube-api-access-5nc42") pod "bc1491f5-7e37-4071-b157-6c88cc14c13d" (UID: "bc1491f5-7e37-4071-b157-6c88cc14c13d"). InnerVolumeSpecName "kube-api-access-5nc42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:03:10.657044 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.656983    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1491f5-7e37-4071-b157-6c88cc14c13d-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:03:10.657044 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.657035    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nc42\" (UniqueName: \"kubernetes.io/projected/bc1491f5-7e37-4071-b157-6c88cc14c13d-kube-api-access-5nc42\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:03:10.657044 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.657045    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:03:10.657044 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.657054    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:03:10.657044 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.657062    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc1491f5-7e37-4071-b157-6c88cc14c13d-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:03:10.965902 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.965863    2575 generic.go:356] "Generic (PLEG): container finished" podID="bc1491f5-7e37-4071-b157-6c88cc14c13d" containerID="94795a3d7281697d74829fffe3da5235dccc594e0831a69e801cd05155463f06" exitCode=0
Sep 09 19:03:10.966118 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.965918    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"
Sep 09 19:03:10.966118 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.965950    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2" event={"ID":"bc1491f5-7e37-4071-b157-6c88cc14c13d","Type":"ContainerDied","Data":"94795a3d7281697d74829fffe3da5235dccc594e0831a69e801cd05155463f06"}
Sep 09 19:03:10.966118 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.965991    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2" event={"ID":"bc1491f5-7e37-4071-b157-6c88cc14c13d","Type":"ContainerDied","Data":"1be7df39630f4c96ac287dd6656b56b74aad3e83e7feebcf402f986d838c0991"}
Sep 09 19:03:10.966118 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.966009    2575 scope.go:117] "RemoveContainer" containerID="94795a3d7281697d74829fffe3da5235dccc594e0831a69e801cd05155463f06"
Sep 09 19:03:10.976916 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.976892    2575 scope.go:117] "RemoveContainer" containerID="94795a3d7281697d74829fffe3da5235dccc594e0831a69e801cd05155463f06"
Sep 09 19:03:10.977272 ip-10-0-134-63 kubenswrapper[2575]: E0909 19:03:10.977251    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94795a3d7281697d74829fffe3da5235dccc594e0831a69e801cd05155463f06\": container with ID starting with 94795a3d7281697d74829fffe3da5235dccc594e0831a69e801cd05155463f06 not found: ID does not exist" containerID="94795a3d7281697d74829fffe3da5235dccc594e0831a69e801cd05155463f06"
Sep 09 19:03:10.977329 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.977283    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94795a3d7281697d74829fffe3da5235dccc594e0831a69e801cd05155463f06"} err="failed to get container status \"94795a3d7281697d74829fffe3da5235dccc594e0831a69e801cd05155463f06\": rpc error: code = NotFound desc = could not find container \"94795a3d7281697d74829fffe3da5235dccc594e0831a69e801cd05155463f06\": container with ID starting with 94795a3d7281697d74829fffe3da5235dccc594e0831a69e801cd05155463f06 not found: ID does not exist"
Sep 09 19:03:10.990194 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.990150    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"]
Sep 09 19:03:10.994146 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:10.994112    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-a63380/ipp-route-a63380-kserve-58998997f5-rnnm2"]
Sep 09 19:03:12.322552 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:12.322491    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc1491f5-7e37-4071-b157-6c88cc14c13d" path="/var/lib/kubelet/pods/bc1491f5-7e37-4071-b157-6c88cc14c13d/volumes"
Sep 09 19:03:44.634667 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.634622    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4"]
Sep 09 19:03:44.635311 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.635289    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb501848-afe9-4546-9b7a-f5358aaa9611" containerName="revoke-keys"
Sep 09 19:03:44.635396 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.635314    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb501848-afe9-4546-9b7a-f5358aaa9611" containerName="revoke-keys"
Sep 09 19:03:44.635396 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.635336    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1491f5-7e37-4071-b157-6c88cc14c13d" containerName="main"
Sep 09 19:03:44.635396 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.635344    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1491f5-7e37-4071-b157-6c88cc14c13d" containerName="main"
Sep 09 19:03:44.635590 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.635444    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb501848-afe9-4546-9b7a-f5358aaa9611" containerName="revoke-keys"
Sep 09 19:03:44.635590 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.635456    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1491f5-7e37-4071-b157-6c88cc14c13d" containerName="main"
Sep 09 19:03:44.638885 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.638860    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4"
Sep 09 19:03:44.641645 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.641623    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 19:03:44.641645 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.641641    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfzt2\""
Sep 09 19:03:44.647729 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.647698    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4"]
Sep 09 19:03:44.718800 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.718760    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc783608-e753-43d7-933f-f6df15542550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4\" (UID: \"bc783608-e753-43d7-933f-f6df15542550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4"
Sep 09 19:03:44.819362 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.819312    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc783608-e753-43d7-933f-f6df15542550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4\" (UID: \"bc783608-e753-43d7-933f-f6df15542550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4"
Sep 09 19:03:44.820041 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.820018    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc783608-e753-43d7-933f-f6df15542550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4\" (UID: \"bc783608-e753-43d7-933f-f6df15542550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4"
Sep 09 19:03:44.951607 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:44.951516    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4"
Sep 09 19:03:45.232282 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:45.232249    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4"]
Sep 09 19:03:45.234619 ip-10-0-134-63 kubenswrapper[2575]: W0909 19:03:45.234590    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc783608_e753_43d7_933f_f6df15542550.slice/crio-0eb23a79965d205d722c814aa8f77da28f159a09d531c58df8611e3650e60085 WatchSource:0}: Error finding container 0eb23a79965d205d722c814aa8f77da28f159a09d531c58df8611e3650e60085: Status 404 returned error can't find the container with id 0eb23a79965d205d722c814aa8f77da28f159a09d531c58df8611e3650e60085
Sep 09 19:03:46.164592 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:46.164460    2575 generic.go:356] "Generic (PLEG): container finished" podID="bc783608-e753-43d7-933f-f6df15542550" containerID="e87c1ff0d4331f0c008e89656b385e539d9a5a7e28b257146f5dbdf232a45f4e" exitCode=0
Sep 09 19:03:46.164592 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:46.164523    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4" event={"ID":"bc783608-e753-43d7-933f-f6df15542550","Type":"ContainerDied","Data":"e87c1ff0d4331f0c008e89656b385e539d9a5a7e28b257146f5dbdf232a45f4e"}
Sep 09 19:03:46.164592 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:46.164589    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4" event={"ID":"bc783608-e753-43d7-933f-f6df15542550","Type":"ContainerStarted","Data":"0eb23a79965d205d722c814aa8f77da28f159a09d531c58df8611e3650e60085"}
Sep 09 19:03:47.312511 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:47.312474    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4"
Sep 09 19:03:47.345741 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:47.345704    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc783608-e753-43d7-933f-f6df15542550-maas-api-service-ca\") pod \"bc783608-e753-43d7-933f-f6df15542550\" (UID: \"bc783608-e753-43d7-933f-f6df15542550\") "
Sep 09 19:03:47.346168 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:47.346133    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc783608-e753-43d7-933f-f6df15542550-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc783608-e753-43d7-933f-f6df15542550" (UID: "bc783608-e753-43d7-933f-f6df15542550"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:03:47.346347 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:47.346326    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc783608-e753-43d7-933f-f6df15542550-maas-api-service-ca\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:03:48.177293 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:48.177261    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4"
Sep 09 19:03:48.177520 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:48.177284    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4" event={"ID":"bc783608-e753-43d7-933f-f6df15542550","Type":"ContainerDied","Data":"0eb23a79965d205d722c814aa8f77da28f159a09d531c58df8611e3650e60085"}
Sep 09 19:03:48.177520 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:48.177367    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eb23a79965d205d722c814aa8f77da28f159a09d531c58df8611e3650e60085"
Sep 09 19:03:52.608359 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:52.608323    2575 scope.go:117] "RemoveContainer" containerID="62f581ec92d61643e21617239a08bccba5b968e3c9b233f730388742f5b36e3b"
Sep 09 19:03:55.232808 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.232770    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"]
Sep 09 19:03:55.233334 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.233170    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst" podUID="edec93d2-7c1a-4f4a-97bf-15477f71133f" containerName="payload-processing" containerID="cri-o://4007fcc7a6dd9b07a6a97a4fa4ce87e0fce6a2bd6b89aa177cf7ecdfa3607f9a" gracePeriod=30
Sep 09 19:03:55.236511 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.236459    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"]
Sep 09 19:03:55.236973 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.236939    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6" podUID="ca2ec485-7c66-4dde-8cdf-59daf91a6c86" containerName="payload-pre-processing" containerID="cri-o://8aef7b95b3c4d9f060e2daebc39323509fe6e5f28e404f7546a74ed29f6eef4f" gracePeriod=30
Sep 09 19:03:55.391526 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.391472    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4"]
Sep 09 19:03:55.394101 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.394067    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-03128a-t6kw4"]
Sep 09 19:03:55.525253 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.525224    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"
Sep 09 19:03:55.532777 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.532744    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"
Sep 09 19:03:55.634122 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.634087    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2ec485-7c66-4dde-8cdf-59daf91a6c86-plugins-config-volume\") pod \"ca2ec485-7c66-4dde-8cdf-59daf91a6c86\" (UID: \"ca2ec485-7c66-4dde-8cdf-59daf91a6c86\") "
Sep 09 19:03:55.634122 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.634133    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjzxx\" (UniqueName: \"kubernetes.io/projected/edec93d2-7c1a-4f4a-97bf-15477f71133f-kube-api-access-wjzxx\") pod \"edec93d2-7c1a-4f4a-97bf-15477f71133f\" (UID: \"edec93d2-7c1a-4f4a-97bf-15477f71133f\") "
Sep 09 19:03:55.634393 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.634167    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dc6x\" (UniqueName: \"kubernetes.io/projected/ca2ec485-7c66-4dde-8cdf-59daf91a6c86-kube-api-access-4dc6x\") pod \"ca2ec485-7c66-4dde-8cdf-59daf91a6c86\" (UID: \"ca2ec485-7c66-4dde-8cdf-59daf91a6c86\") "
Sep 09 19:03:55.634393 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.634199    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edec93d2-7c1a-4f4a-97bf-15477f71133f-plugins-config-volume\") pod \"edec93d2-7c1a-4f4a-97bf-15477f71133f\" (UID: \"edec93d2-7c1a-4f4a-97bf-15477f71133f\") "
Sep 09 19:03:55.634541 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.634512    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2ec485-7c66-4dde-8cdf-59daf91a6c86-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca2ec485-7c66-4dde-8cdf-59daf91a6c86" (UID: "ca2ec485-7c66-4dde-8cdf-59daf91a6c86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:03:55.634669 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.634647    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edec93d2-7c1a-4f4a-97bf-15477f71133f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "edec93d2-7c1a-4f4a-97bf-15477f71133f" (UID: "edec93d2-7c1a-4f4a-97bf-15477f71133f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:03:55.636425 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.636399    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edec93d2-7c1a-4f4a-97bf-15477f71133f-kube-api-access-wjzxx" (OuterVolumeSpecName: "kube-api-access-wjzxx") pod "edec93d2-7c1a-4f4a-97bf-15477f71133f" (UID: "edec93d2-7c1a-4f4a-97bf-15477f71133f"). InnerVolumeSpecName "kube-api-access-wjzxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:03:55.636529 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.636433    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2ec485-7c66-4dde-8cdf-59daf91a6c86-kube-api-access-4dc6x" (OuterVolumeSpecName: "kube-api-access-4dc6x") pod "ca2ec485-7c66-4dde-8cdf-59daf91a6c86" (UID: "ca2ec485-7c66-4dde-8cdf-59daf91a6c86"). InnerVolumeSpecName "kube-api-access-4dc6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:03:55.736010 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.735955    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2ec485-7c66-4dde-8cdf-59daf91a6c86-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:03:55.736010 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.735999    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjzxx\" (UniqueName: \"kubernetes.io/projected/edec93d2-7c1a-4f4a-97bf-15477f71133f-kube-api-access-wjzxx\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:03:55.736010 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.736008    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dc6x\" (UniqueName: \"kubernetes.io/projected/ca2ec485-7c66-4dde-8cdf-59daf91a6c86-kube-api-access-4dc6x\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:03:55.736010 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:55.736017    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edec93d2-7c1a-4f4a-97bf-15477f71133f-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:03:56.224949 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.224909    2575 generic.go:356] "Generic (PLEG): container finished" podID="edec93d2-7c1a-4f4a-97bf-15477f71133f" containerID="4007fcc7a6dd9b07a6a97a4fa4ce87e0fce6a2bd6b89aa177cf7ecdfa3607f9a" exitCode=0
Sep 09 19:03:56.225144 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.224960    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"
Sep 09 19:03:56.225144 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.224989    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst" event={"ID":"edec93d2-7c1a-4f4a-97bf-15477f71133f","Type":"ContainerDied","Data":"4007fcc7a6dd9b07a6a97a4fa4ce87e0fce6a2bd6b89aa177cf7ecdfa3607f9a"}
Sep 09 19:03:56.225144 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.225049    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst" event={"ID":"edec93d2-7c1a-4f4a-97bf-15477f71133f","Type":"ContainerDied","Data":"e204ea6efb4d4938158a50d4591f729c43fdfeff25060e0fc112cbc445bdc2f8"}
Sep 09 19:03:56.225144 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.225073    2575 scope.go:117] "RemoveContainer" containerID="4007fcc7a6dd9b07a6a97a4fa4ce87e0fce6a2bd6b89aa177cf7ecdfa3607f9a"
Sep 09 19:03:56.226621 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.226596    2575 generic.go:356] "Generic (PLEG): container finished" podID="ca2ec485-7c66-4dde-8cdf-59daf91a6c86" containerID="8aef7b95b3c4d9f060e2daebc39323509fe6e5f28e404f7546a74ed29f6eef4f" exitCode=0
Sep 09 19:03:56.226748 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.226630    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6" event={"ID":"ca2ec485-7c66-4dde-8cdf-59daf91a6c86","Type":"ContainerDied","Data":"8aef7b95b3c4d9f060e2daebc39323509fe6e5f28e404f7546a74ed29f6eef4f"}
Sep 09 19:03:56.226748 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.226650    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"
Sep 09 19:03:56.226748 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.226692    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6" event={"ID":"ca2ec485-7c66-4dde-8cdf-59daf91a6c86","Type":"ContainerDied","Data":"8276e270ada137c3ac2898f8fb595c2e462e883133cce577d9f747270514d2aa"}
Sep 09 19:03:56.237399 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.237211    2575 scope.go:117] "RemoveContainer" containerID="4007fcc7a6dd9b07a6a97a4fa4ce87e0fce6a2bd6b89aa177cf7ecdfa3607f9a"
Sep 09 19:03:56.237693 ip-10-0-134-63 kubenswrapper[2575]: E0909 19:03:56.237662    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4007fcc7a6dd9b07a6a97a4fa4ce87e0fce6a2bd6b89aa177cf7ecdfa3607f9a\": container with ID starting with 4007fcc7a6dd9b07a6a97a4fa4ce87e0fce6a2bd6b89aa177cf7ecdfa3607f9a not found: ID does not exist" containerID="4007fcc7a6dd9b07a6a97a4fa4ce87e0fce6a2bd6b89aa177cf7ecdfa3607f9a"
Sep 09 19:03:56.237733 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.237697    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4007fcc7a6dd9b07a6a97a4fa4ce87e0fce6a2bd6b89aa177cf7ecdfa3607f9a"} err="failed to get container status \"4007fcc7a6dd9b07a6a97a4fa4ce87e0fce6a2bd6b89aa177cf7ecdfa3607f9a\": rpc error: code = NotFound desc = could not find container \"4007fcc7a6dd9b07a6a97a4fa4ce87e0fce6a2bd6b89aa177cf7ecdfa3607f9a\": container with ID starting with 4007fcc7a6dd9b07a6a97a4fa4ce87e0fce6a2bd6b89aa177cf7ecdfa3607f9a not found: ID does not exist"
Sep 09 19:03:56.237733 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.237717    2575 scope.go:117] "RemoveContainer" containerID="8aef7b95b3c4d9f060e2daebc39323509fe6e5f28e404f7546a74ed29f6eef4f"
Sep 09 19:03:56.248931 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.248905    2575 scope.go:117] "RemoveContainer" containerID="8aef7b95b3c4d9f060e2daebc39323509fe6e5f28e404f7546a74ed29f6eef4f"
Sep 09 19:03:56.249346 ip-10-0-134-63 kubenswrapper[2575]: E0909 19:03:56.249319    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aef7b95b3c4d9f060e2daebc39323509fe6e5f28e404f7546a74ed29f6eef4f\": container with ID starting with 8aef7b95b3c4d9f060e2daebc39323509fe6e5f28e404f7546a74ed29f6eef4f not found: ID does not exist" containerID="8aef7b95b3c4d9f060e2daebc39323509fe6e5f28e404f7546a74ed29f6eef4f"
Sep 09 19:03:56.249408 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.249360    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aef7b95b3c4d9f060e2daebc39323509fe6e5f28e404f7546a74ed29f6eef4f"} err="failed to get container status \"8aef7b95b3c4d9f060e2daebc39323509fe6e5f28e404f7546a74ed29f6eef4f\": rpc error: code = NotFound desc = could not find container \"8aef7b95b3c4d9f060e2daebc39323509fe6e5f28e404f7546a74ed29f6eef4f\": container with ID starting with 8aef7b95b3c4d9f060e2daebc39323509fe6e5f28e404f7546a74ed29f6eef4f not found: ID does not exist"
Sep 09 19:03:56.250652 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.250630    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"]
Sep 09 19:03:56.254474 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.254443    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-03128a-789f459677-frqst"]
Sep 09 19:03:56.267112 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.267067    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"]
Sep 09 19:03:56.273571 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.273534    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-03128a-7bd97ff894-2kwt6"]
Sep 09 19:03:56.321332 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.321293    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc783608-e753-43d7-933f-f6df15542550" path="/var/lib/kubelet/pods/bc783608-e753-43d7-933f-f6df15542550/volumes"
Sep 09 19:03:56.321750 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.321734    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca2ec485-7c66-4dde-8cdf-59daf91a6c86" path="/var/lib/kubelet/pods/ca2ec485-7c66-4dde-8cdf-59daf91a6c86/volumes"
Sep 09 19:03:56.322073 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:03:56.322060    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edec93d2-7c1a-4f4a-97bf-15477f71133f" path="/var/lib/kubelet/pods/edec93d2-7c1a-4f4a-97bf-15477f71133f/volumes"
Sep 09 19:04:12.676756 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:12.676715    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-282de3-29816340-kkskr"]
Sep 09 19:04:12.679491 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:12.679460    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-282de3-29816340-kkskr"]
Sep 09 19:04:12.702340 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:12.702299    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"]
Sep 09 19:04:12.702633 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:12.702596    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn" podUID="bce8222c-e784-4418-be86-5ba581e970b7" containerName="payload-processing" containerID="cri-o://25869d1a1feb63e7b7f3381fa53860afa2f4bf5f4e25f39d8f663377a37105ab" gracePeriod=30
Sep 09 19:04:12.711549 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:12.710090    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"]
Sep 09 19:04:12.711549 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:12.710332    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm" podUID="8820e118-7a7c-4102-a4cd-b0f52f52d4b4" containerName="payload-pre-processing" containerID="cri-o://dc49265753f1c1d63b6e0e18e780836302cc6796e0ac20e8a11117ba5838cb3f" gracePeriod=30
Sep 09 19:04:12.995185 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:12.995156    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"
Sep 09 19:04:12.998388 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:12.998365    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"
Sep 09 19:04:13.013538 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.013486    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46nm7\" (UniqueName: \"kubernetes.io/projected/8820e118-7a7c-4102-a4cd-b0f52f52d4b4-kube-api-access-46nm7\") pod \"8820e118-7a7c-4102-a4cd-b0f52f52d4b4\" (UID: \"8820e118-7a7c-4102-a4cd-b0f52f52d4b4\") "
Sep 09 19:04:13.013717 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.013604    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce8222c-e784-4418-be86-5ba581e970b7-plugins-config-volume\") pod \"bce8222c-e784-4418-be86-5ba581e970b7\" (UID: \"bce8222c-e784-4418-be86-5ba581e970b7\") "
Sep 09 19:04:13.013717 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.013665    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8820e118-7a7c-4102-a4cd-b0f52f52d4b4-plugins-config-volume\") pod \"8820e118-7a7c-4102-a4cd-b0f52f52d4b4\" (UID: \"8820e118-7a7c-4102-a4cd-b0f52f52d4b4\") "
Sep 09 19:04:13.013835 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.013718    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k5hd\" (UniqueName: \"kubernetes.io/projected/bce8222c-e784-4418-be86-5ba581e970b7-kube-api-access-6k5hd\") pod \"bce8222c-e784-4418-be86-5ba581e970b7\" (UID: \"bce8222c-e784-4418-be86-5ba581e970b7\") "
Sep 09 19:04:13.014374 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.014321    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce8222c-e784-4418-be86-5ba581e970b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bce8222c-e784-4418-be86-5ba581e970b7" (UID: "bce8222c-e784-4418-be86-5ba581e970b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:04:13.014494 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.014443    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8820e118-7a7c-4102-a4cd-b0f52f52d4b4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8820e118-7a7c-4102-a4cd-b0f52f52d4b4" (UID: "8820e118-7a7c-4102-a4cd-b0f52f52d4b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:04:13.016827 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.016798    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce8222c-e784-4418-be86-5ba581e970b7-kube-api-access-6k5hd" (OuterVolumeSpecName: "kube-api-access-6k5hd") pod "bce8222c-e784-4418-be86-5ba581e970b7" (UID: "bce8222c-e784-4418-be86-5ba581e970b7"). InnerVolumeSpecName "kube-api-access-6k5hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:04:13.016980 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.016951    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8820e118-7a7c-4102-a4cd-b0f52f52d4b4-kube-api-access-46nm7" (OuterVolumeSpecName: "kube-api-access-46nm7") pod "8820e118-7a7c-4102-a4cd-b0f52f52d4b4" (UID: "8820e118-7a7c-4102-a4cd-b0f52f52d4b4"). InnerVolumeSpecName "kube-api-access-46nm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:04:13.114721 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.114684    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46nm7\" (UniqueName: \"kubernetes.io/projected/8820e118-7a7c-4102-a4cd-b0f52f52d4b4-kube-api-access-46nm7\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:04:13.114721 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.114716    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce8222c-e784-4418-be86-5ba581e970b7-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:04:13.114721 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.114728    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8820e118-7a7c-4102-a4cd-b0f52f52d4b4-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:04:13.114979 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.114738    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k5hd\" (UniqueName: \"kubernetes.io/projected/bce8222c-e784-4418-be86-5ba581e970b7-kube-api-access-6k5hd\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:04:13.327150 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.327042    2575 generic.go:356] "Generic (PLEG): container finished" podID="bce8222c-e784-4418-be86-5ba581e970b7" containerID="25869d1a1feb63e7b7f3381fa53860afa2f4bf5f4e25f39d8f663377a37105ab" exitCode=0
Sep 09 19:04:13.327150 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.327092    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"
Sep 09 19:04:13.327150 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.327123    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn" event={"ID":"bce8222c-e784-4418-be86-5ba581e970b7","Type":"ContainerDied","Data":"25869d1a1feb63e7b7f3381fa53860afa2f4bf5f4e25f39d8f663377a37105ab"}
Sep 09 19:04:13.327427 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.327175    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn" event={"ID":"bce8222c-e784-4418-be86-5ba581e970b7","Type":"ContainerDied","Data":"bc068ca137e96014aaec9cc9afe30ae8ceabfe785c79a1356ed6c76eb6967fd6"}
Sep 09 19:04:13.327427 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.327206    2575 scope.go:117] "RemoveContainer" containerID="25869d1a1feb63e7b7f3381fa53860afa2f4bf5f4e25f39d8f663377a37105ab"
Sep 09 19:04:13.328692 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.328669    2575 generic.go:356] "Generic (PLEG): container finished" podID="8820e118-7a7c-4102-a4cd-b0f52f52d4b4" containerID="dc49265753f1c1d63b6e0e18e780836302cc6796e0ac20e8a11117ba5838cb3f" exitCode=0
Sep 09 19:04:13.328817 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.328705    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm" event={"ID":"8820e118-7a7c-4102-a4cd-b0f52f52d4b4","Type":"ContainerDied","Data":"dc49265753f1c1d63b6e0e18e780836302cc6796e0ac20e8a11117ba5838cb3f"}
Sep 09 19:04:13.328817 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.328718    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"
Sep 09 19:04:13.328817 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.328754    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm" event={"ID":"8820e118-7a7c-4102-a4cd-b0f52f52d4b4","Type":"ContainerDied","Data":"dde28c3353dee7633d442fe0d510c70e064ecc09792d0951ba6dbf11a0709f64"}
Sep 09 19:04:13.336921 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.336899    2575 scope.go:117] "RemoveContainer" containerID="25869d1a1feb63e7b7f3381fa53860afa2f4bf5f4e25f39d8f663377a37105ab"
Sep 09 19:04:13.337237 ip-10-0-134-63 kubenswrapper[2575]: E0909 19:04:13.337216    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25869d1a1feb63e7b7f3381fa53860afa2f4bf5f4e25f39d8f663377a37105ab\": container with ID starting with 25869d1a1feb63e7b7f3381fa53860afa2f4bf5f4e25f39d8f663377a37105ab not found: ID does not exist" containerID="25869d1a1feb63e7b7f3381fa53860afa2f4bf5f4e25f39d8f663377a37105ab"
Sep 09 19:04:13.337319 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.337245    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25869d1a1feb63e7b7f3381fa53860afa2f4bf5f4e25f39d8f663377a37105ab"} err="failed to get container status \"25869d1a1feb63e7b7f3381fa53860afa2f4bf5f4e25f39d8f663377a37105ab\": rpc error: code = NotFound desc = could not find container \"25869d1a1feb63e7b7f3381fa53860afa2f4bf5f4e25f39d8f663377a37105ab\": container with ID starting with 25869d1a1feb63e7b7f3381fa53860afa2f4bf5f4e25f39d8f663377a37105ab not found: ID does not exist"
Sep 09 19:04:13.337319 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.337262    2575 scope.go:117] "RemoveContainer" containerID="dc49265753f1c1d63b6e0e18e780836302cc6796e0ac20e8a11117ba5838cb3f"
Sep 09 19:04:13.346518 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.346485    2575 scope.go:117] "RemoveContainer" containerID="dc49265753f1c1d63b6e0e18e780836302cc6796e0ac20e8a11117ba5838cb3f"
Sep 09 19:04:13.346836 ip-10-0-134-63 kubenswrapper[2575]: E0909 19:04:13.346812    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc49265753f1c1d63b6e0e18e780836302cc6796e0ac20e8a11117ba5838cb3f\": container with ID starting with dc49265753f1c1d63b6e0e18e780836302cc6796e0ac20e8a11117ba5838cb3f not found: ID does not exist" containerID="dc49265753f1c1d63b6e0e18e780836302cc6796e0ac20e8a11117ba5838cb3f"
Sep 09 19:04:13.346912 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.346845    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc49265753f1c1d63b6e0e18e780836302cc6796e0ac20e8a11117ba5838cb3f"} err="failed to get container status \"dc49265753f1c1d63b6e0e18e780836302cc6796e0ac20e8a11117ba5838cb3f\": rpc error: code = NotFound desc = could not find container \"dc49265753f1c1d63b6e0e18e780836302cc6796e0ac20e8a11117ba5838cb3f\": container with ID starting with dc49265753f1c1d63b6e0e18e780836302cc6796e0ac20e8a11117ba5838cb3f not found: ID does not exist"
Sep 09 19:04:13.354867 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.354835    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"]
Sep 09 19:04:13.357640 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.357615    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-282de3-59c479d8f8-fdckm"]
Sep 09 19:04:13.368179 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.368144    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"]
Sep 09 19:04:13.372146 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:13.372117    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-282de3-5d78f4d587-kbtvn"]
Sep 09 19:04:14.321358 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:14.321322    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ecfe888-50ff-4437-9f5d-4a7b73c1f748" path="/var/lib/kubelet/pods/3ecfe888-50ff-4437-9f5d-4a7b73c1f748/volumes"
Sep 09 19:04:14.321782 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:14.321748    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8820e118-7a7c-4102-a4cd-b0f52f52d4b4" path="/var/lib/kubelet/pods/8820e118-7a7c-4102-a4cd-b0f52f52d4b4/volumes"
Sep 09 19:04:14.322089 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:14.322075    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bce8222c-e784-4418-be86-5ba581e970b7" path="/var/lib/kubelet/pods/bce8222c-e784-4418-be86-5ba581e970b7/volumes"
Sep 09 19:04:30.401824 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.401756    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e012d-29816340-wxzz9"]
Sep 09 19:04:30.405014 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.404979    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e012d-29816340-wxzz9"]
Sep 09 19:04:30.426599 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.426565    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"]
Sep 09 19:04:30.426840 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.426794    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9" podUID="15a88570-392e-4cd0-9528-c541202f5a70" containerName="payload-processing" containerID="cri-o://c56ca9cc89986d618050968d9337f5fdab49853923473a7568e3e0e03e972557" gracePeriod=30
Sep 09 19:04:30.433432 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.433401    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"]
Sep 09 19:04:30.433687 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.433654    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps" podUID="b2c47ad7-8b27-4e6b-82fd-8c754b922373" containerName="payload-pre-processing" containerID="cri-o://4116f5916dda37323092bdd0cbd45ad9ec7cf3e7018a5a0e58d47428ce4a4d70" gracePeriod=30
Sep 09 19:04:30.709136 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.709110    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"
Sep 09 19:04:30.712197 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.712172    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"
Sep 09 19:04:30.787734 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.787689    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a88570-392e-4cd0-9528-c541202f5a70-plugins-config-volume\") pod \"15a88570-392e-4cd0-9528-c541202f5a70\" (UID: \"15a88570-392e-4cd0-9528-c541202f5a70\") "
Sep 09 19:04:30.787937 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.787830    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llq4r\" (UniqueName: \"kubernetes.io/projected/15a88570-392e-4cd0-9528-c541202f5a70-kube-api-access-llq4r\") pod \"15a88570-392e-4cd0-9528-c541202f5a70\" (UID: \"15a88570-392e-4cd0-9528-c541202f5a70\") "
Sep 09 19:04:30.787937 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.787863    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c47ad7-8b27-4e6b-82fd-8c754b922373-plugins-config-volume\") pod \"b2c47ad7-8b27-4e6b-82fd-8c754b922373\" (UID: \"b2c47ad7-8b27-4e6b-82fd-8c754b922373\") "
Sep 09 19:04:30.787937 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.787904    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc7dn\" (UniqueName: \"kubernetes.io/projected/b2c47ad7-8b27-4e6b-82fd-8c754b922373-kube-api-access-rc7dn\") pod \"b2c47ad7-8b27-4e6b-82fd-8c754b922373\" (UID: \"b2c47ad7-8b27-4e6b-82fd-8c754b922373\") "
Sep 09 19:04:30.788158 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.788125    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a88570-392e-4cd0-9528-c541202f5a70-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15a88570-392e-4cd0-9528-c541202f5a70" (UID: "15a88570-392e-4cd0-9528-c541202f5a70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:04:30.788304 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.788275    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c47ad7-8b27-4e6b-82fd-8c754b922373-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b2c47ad7-8b27-4e6b-82fd-8c754b922373" (UID: "b2c47ad7-8b27-4e6b-82fd-8c754b922373"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:04:30.790071 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.790045    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c47ad7-8b27-4e6b-82fd-8c754b922373-kube-api-access-rc7dn" (OuterVolumeSpecName: "kube-api-access-rc7dn") pod "b2c47ad7-8b27-4e6b-82fd-8c754b922373" (UID: "b2c47ad7-8b27-4e6b-82fd-8c754b922373"). InnerVolumeSpecName "kube-api-access-rc7dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:04:30.790178 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.790089    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a88570-392e-4cd0-9528-c541202f5a70-kube-api-access-llq4r" (OuterVolumeSpecName: "kube-api-access-llq4r") pod "15a88570-392e-4cd0-9528-c541202f5a70" (UID: "15a88570-392e-4cd0-9528-c541202f5a70"). InnerVolumeSpecName "kube-api-access-llq4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:04:30.889099 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.889055    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llq4r\" (UniqueName: \"kubernetes.io/projected/15a88570-392e-4cd0-9528-c541202f5a70-kube-api-access-llq4r\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:04:30.889099 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.889087    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2c47ad7-8b27-4e6b-82fd-8c754b922373-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:04:30.889099 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.889096    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc7dn\" (UniqueName: \"kubernetes.io/projected/b2c47ad7-8b27-4e6b-82fd-8c754b922373-kube-api-access-rc7dn\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:04:30.889099 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:30.889106    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15a88570-392e-4cd0-9528-c541202f5a70-plugins-config-volume\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:04:31.435657 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.435622    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2c47ad7-8b27-4e6b-82fd-8c754b922373" containerID="4116f5916dda37323092bdd0cbd45ad9ec7cf3e7018a5a0e58d47428ce4a4d70" exitCode=0
Sep 09 19:04:31.436146 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.435663    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"
Sep 09 19:04:31.436146 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.435696    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps" event={"ID":"b2c47ad7-8b27-4e6b-82fd-8c754b922373","Type":"ContainerDied","Data":"4116f5916dda37323092bdd0cbd45ad9ec7cf3e7018a5a0e58d47428ce4a4d70"}
Sep 09 19:04:31.436146 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.435759    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps" event={"ID":"b2c47ad7-8b27-4e6b-82fd-8c754b922373","Type":"ContainerDied","Data":"754012caddad17dba80ea22ee5d13035b27645a17b417355541a56683b6c847f"}
Sep 09 19:04:31.436146 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.435787    2575 scope.go:117] "RemoveContainer" containerID="4116f5916dda37323092bdd0cbd45ad9ec7cf3e7018a5a0e58d47428ce4a4d70"
Sep 09 19:04:31.437321 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.437298    2575 generic.go:356] "Generic (PLEG): container finished" podID="15a88570-392e-4cd0-9528-c541202f5a70" containerID="c56ca9cc89986d618050968d9337f5fdab49853923473a7568e3e0e03e972557" exitCode=0
Sep 09 19:04:31.437418 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.437345    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"
Sep 09 19:04:31.437418 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.437398    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9" event={"ID":"15a88570-392e-4cd0-9528-c541202f5a70","Type":"ContainerDied","Data":"c56ca9cc89986d618050968d9337f5fdab49853923473a7568e3e0e03e972557"}
Sep 09 19:04:31.437527 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.437445    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9" event={"ID":"15a88570-392e-4cd0-9528-c541202f5a70","Type":"ContainerDied","Data":"832b55892da4d304f9f0e560962cf8497bd1951d7bbcd37733a72de54180c92e"}
Sep 09 19:04:31.450274 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.450256    2575 scope.go:117] "RemoveContainer" containerID="4116f5916dda37323092bdd0cbd45ad9ec7cf3e7018a5a0e58d47428ce4a4d70"
Sep 09 19:04:31.450636 ip-10-0-134-63 kubenswrapper[2575]: E0909 19:04:31.450613    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4116f5916dda37323092bdd0cbd45ad9ec7cf3e7018a5a0e58d47428ce4a4d70\": container with ID starting with 4116f5916dda37323092bdd0cbd45ad9ec7cf3e7018a5a0e58d47428ce4a4d70 not found: ID does not exist" containerID="4116f5916dda37323092bdd0cbd45ad9ec7cf3e7018a5a0e58d47428ce4a4d70"
Sep 09 19:04:31.450746 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.450647    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4116f5916dda37323092bdd0cbd45ad9ec7cf3e7018a5a0e58d47428ce4a4d70"} err="failed to get container status \"4116f5916dda37323092bdd0cbd45ad9ec7cf3e7018a5a0e58d47428ce4a4d70\": rpc error: code = NotFound desc = could not find container \"4116f5916dda37323092bdd0cbd45ad9ec7cf3e7018a5a0e58d47428ce4a4d70\": container with ID starting with 4116f5916dda37323092bdd0cbd45ad9ec7cf3e7018a5a0e58d47428ce4a4d70 not found: ID does not exist"
Sep 09 19:04:31.450746 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.450664    2575 scope.go:117] "RemoveContainer" containerID="c56ca9cc89986d618050968d9337f5fdab49853923473a7568e3e0e03e972557"
Sep 09 19:04:31.460184 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.460158    2575 scope.go:117] "RemoveContainer" containerID="c56ca9cc89986d618050968d9337f5fdab49853923473a7568e3e0e03e972557"
Sep 09 19:04:31.460638 ip-10-0-134-63 kubenswrapper[2575]: E0909 19:04:31.460612    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c56ca9cc89986d618050968d9337f5fdab49853923473a7568e3e0e03e972557\": container with ID starting with c56ca9cc89986d618050968d9337f5fdab49853923473a7568e3e0e03e972557 not found: ID does not exist" containerID="c56ca9cc89986d618050968d9337f5fdab49853923473a7568e3e0e03e972557"
Sep 09 19:04:31.460742 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.460648    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c56ca9cc89986d618050968d9337f5fdab49853923473a7568e3e0e03e972557"} err="failed to get container status \"c56ca9cc89986d618050968d9337f5fdab49853923473a7568e3e0e03e972557\": rpc error: code = NotFound desc = could not find container \"c56ca9cc89986d618050968d9337f5fdab49853923473a7568e3e0e03e972557\": container with ID starting with c56ca9cc89986d618050968d9337f5fdab49853923473a7568e3e0e03e972557 not found: ID does not exist"
Sep 09 19:04:31.462400 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.462376    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"]
Sep 09 19:04:31.464433 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.464410    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e012d-646f77b84f-jvdt9"]
Sep 09 19:04:31.474577 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.474544    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"]
Sep 09 19:04:31.478397 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:31.478365    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e012d-5b49d77994-4d9ps"]
Sep 09 19:04:32.321179 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:32.321147    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15a88570-392e-4cd0-9528-c541202f5a70" path="/var/lib/kubelet/pods/15a88570-392e-4cd0-9528-c541202f5a70/volumes"
Sep 09 19:04:32.321566 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:32.321552    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2c47ad7-8b27-4e6b-82fd-8c754b922373" path="/var/lib/kubelet/pods/b2c47ad7-8b27-4e6b-82fd-8c754b922373/volumes"
Sep 09 19:04:32.321816 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:04:32.321804    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cda04f77-87e7-43d7-8091-dfbc03cf0199" path="/var/lib/kubelet/pods/cda04f77-87e7-43d7-8091-dfbc03cf0199/volumes"
Sep 09 19:05:52.669182 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:05:52.669144    2575 scope.go:117] "RemoveContainer" containerID="0717ce6127a0e56cb78a187908d8737f7d562efa84f465c45c35bcacb4f4bced"
Sep 09 19:05:52.677940 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:05:52.677918    2575 scope.go:117] "RemoveContainer" containerID="8ad14bd3bbcbec8bcec1a46f605b2d9e6b06ba94bd3d1dcf2a07f64c5eba7031"
Sep 09 19:06:52.417658 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:06:52.417625    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 19:06:52.425711 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:06:52.425684    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 19:06:52.711050 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:06:52.710959    2575 scope.go:117] "RemoveContainer" containerID="1947209010c04572ed3adc65dfadbc340d904f1b7c7011f6c32a42dc3033d087"
Sep 09 19:06:52.720157 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:06:52.720133    2575 scope.go:117] "RemoveContainer" containerID="15d56487e758919ae6da9b77255a7a2bdc269fee1bde12bba93f6b0fe739ecec"
Sep 09 19:08:41.544411 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544371    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"]
Sep 09 19:08:41.544883 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544815    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2ec485-7c66-4dde-8cdf-59daf91a6c86" containerName="payload-pre-processing"
Sep 09 19:08:41.544883 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544828    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2ec485-7c66-4dde-8cdf-59daf91a6c86" containerName="payload-pre-processing"
Sep 09 19:08:41.544883 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544843    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c47ad7-8b27-4e6b-82fd-8c754b922373" containerName="payload-pre-processing"
Sep 09 19:08:41.544883 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544849    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c47ad7-8b27-4e6b-82fd-8c754b922373" containerName="payload-pre-processing"
Sep 09 19:08:41.544883 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544856    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edec93d2-7c1a-4f4a-97bf-15477f71133f" containerName="payload-processing"
Sep 09 19:08:41.544883 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544862    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="edec93d2-7c1a-4f4a-97bf-15477f71133f" containerName="payload-processing"
Sep 09 19:08:41.544883 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544873    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a88570-392e-4cd0-9528-c541202f5a70" containerName="payload-processing"
Sep 09 19:08:41.544883 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544878    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a88570-392e-4cd0-9528-c541202f5a70" containerName="payload-processing"
Sep 09 19:08:41.544883 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544887    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc783608-e753-43d7-933f-f6df15542550" containerName="revoke-keys"
Sep 09 19:08:41.545164 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544894    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc783608-e753-43d7-933f-f6df15542550" containerName="revoke-keys"
Sep 09 19:08:41.545164 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544901    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8820e118-7a7c-4102-a4cd-b0f52f52d4b4" containerName="payload-pre-processing"
Sep 09 19:08:41.545164 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544907    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8820e118-7a7c-4102-a4cd-b0f52f52d4b4" containerName="payload-pre-processing"
Sep 09 19:08:41.545164 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544918    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce8222c-e784-4418-be86-5ba581e970b7" containerName="payload-processing"
Sep 09 19:08:41.545164 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544923    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce8222c-e784-4418-be86-5ba581e970b7" containerName="payload-processing"
Sep 09 19:08:41.545164 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544984    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8820e118-7a7c-4102-a4cd-b0f52f52d4b4" containerName="payload-pre-processing"
Sep 09 19:08:41.545164 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544990    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2c47ad7-8b27-4e6b-82fd-8c754b922373" containerName="payload-pre-processing"
Sep 09 19:08:41.545164 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.544999    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca2ec485-7c66-4dde-8cdf-59daf91a6c86" containerName="payload-pre-processing"
Sep 09 19:08:41.545164 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.545007    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="edec93d2-7c1a-4f4a-97bf-15477f71133f" containerName="payload-processing"
Sep 09 19:08:41.545164 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.545017    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a88570-392e-4cd0-9528-c541202f5a70" containerName="payload-processing"
Sep 09 19:08:41.545164 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.545024    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce8222c-e784-4418-be86-5ba581e970b7" containerName="payload-processing"
Sep 09 19:08:41.545164 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.545035    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc783608-e753-43d7-933f-f6df15542550" containerName="revoke-keys"
Sep 09 19:08:41.548154 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.548131    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.550873 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.550847    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9cde296a-kserve-self-signed-certs\""
Sep 09 19:08:41.560554 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.560519    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"]
Sep 09 19:08:41.639583 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.639531    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-dshm\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.639790 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.639595    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-home\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.639790 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.639668    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-model-cache\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.639790 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.639750    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cb5b9a-275a-48e8-89db-c7eafd258c30-tls-certs\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.639790 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.639777    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hfxs\" (UniqueName: \"kubernetes.io/projected/f2cb5b9a-275a-48e8-89db-c7eafd258c30-kube-api-access-7hfxs\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.639942 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.639805    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-tmp-dir\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.740449 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.740406    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-home\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.740657 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.740468    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-model-cache\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.740657 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.740556    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cb5b9a-275a-48e8-89db-c7eafd258c30-tls-certs\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.740657 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.740576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hfxs\" (UniqueName: \"kubernetes.io/projected/f2cb5b9a-275a-48e8-89db-c7eafd258c30-kube-api-access-7hfxs\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.740657 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.740601    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-tmp-dir\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.740657 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.740651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-dshm\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.740949 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.740909    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-home\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.740949 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.740917    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-model-cache\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.741058 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.740989    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-tmp-dir\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.742957 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.742937    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-dshm\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.743290 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.743273    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cb5b9a-275a-48e8-89db-c7eafd258c30-tls-certs\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.748790 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.748763    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hfxs\" (UniqueName: \"kubernetes.io/projected/f2cb5b9a-275a-48e8-89db-c7eafd258c30-kube-api-access-7hfxs\") pod \"e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") " pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:41.860043 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:41.860008    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:42.137139 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:42.137108    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"]
Sep 09 19:08:42.139090 ip-10-0-134-63 kubenswrapper[2575]: W0909 19:08:42.139054    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2cb5b9a_275a_48e8_89db_c7eafd258c30.slice/crio-3d2b3496d73eb3916e68399097494e599910a956bd6c4cacdeeab471f2cded70 WatchSource:0}: Error finding container 3d2b3496d73eb3916e68399097494e599910a956bd6c4cacdeeab471f2cded70: Status 404 returned error can't find the container with id 3d2b3496d73eb3916e68399097494e599910a956bd6c4cacdeeab471f2cded70
Sep 09 19:08:42.785140 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:42.785096    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj" event={"ID":"f2cb5b9a-275a-48e8-89db-c7eafd258c30","Type":"ContainerStarted","Data":"45bb48404538a287792d67d9a9833cd53a3196ead14e08d71020989d773b494c"}
Sep 09 19:08:42.785140 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:42.785146    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj" event={"ID":"f2cb5b9a-275a-48e8-89db-c7eafd258c30","Type":"ContainerStarted","Data":"3d2b3496d73eb3916e68399097494e599910a956bd6c4cacdeeab471f2cded70"}
Sep 09 19:08:42.803163 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:42.803104    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj" podStartSLOduration=1.80308807 podStartE2EDuration="1.80308807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:08:42.801729261 +0000 UTC m=+2211.039099077" watchObservedRunningTime="2026-09-09 19:08:42.80308807 +0000 UTC m=+2211.040457884"
Sep 09 19:08:51.860958 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:51.860909    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:51.860958 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:51.860963    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:51.873451 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:51.873419    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:52.752419 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:52.752381    2575 scope.go:117] "RemoveContainer" containerID="ffc66a0a3fe55863d76dc7abb6a6b6f55438338ced9c43d78459e05768732955"
Sep 09 19:08:52.767654 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:52.767622    2575 scope.go:117] "RemoveContainer" containerID="cfdb836dea9be36795a6fde0413d0ac479938fd472346058f7cc9bd05778eadf"
Sep 09 19:08:52.851125 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:52.851088    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:08:52.947702 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:52.947663    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"]
Sep 09 19:08:52.953487 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:52.953457    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:52.956008 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:52.955980    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-9cde296a-kserve-self-signed-certs\""
Sep 09 19:08:52.959804 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:52.959771    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"]
Sep 09 19:08:53.057749 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.057660    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-tmp-dir\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.057749 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.057715    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6m2w\" (UniqueName: \"kubernetes.io/projected/fe7e6540-6323-49bb-be3a-a49687822767-kube-api-access-x6m2w\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.057951 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.057772    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-home\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.057951 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.057836    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7e6540-6323-49bb-be3a-a49687822767-tls-certs\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.057951 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.057862    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-model-cache\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.057951 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.057907    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-dshm\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.158722 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.158680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7e6540-6323-49bb-be3a-a49687822767-tls-certs\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.158722 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.158725    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-model-cache\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.159002 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.158746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-dshm\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.159002 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.158792    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-tmp-dir\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.159002 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.158816    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6m2w\" (UniqueName: \"kubernetes.io/projected/fe7e6540-6323-49bb-be3a-a49687822767-kube-api-access-x6m2w\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.159002 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.158839    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-home\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.159205 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.159144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-model-cache\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.159273 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.159247    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-tmp-dir\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.159311 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.159284    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-home\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.161229 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.161202    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-dshm\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.161532 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.161493    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7e6540-6323-49bb-be3a-a49687822767-tls-certs\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.167449 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.167423    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6m2w\" (UniqueName: \"kubernetes.io/projected/fe7e6540-6323-49bb-be3a-a49687822767-kube-api-access-x6m2w\") pod \"e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") " pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.265383 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.265336    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:08:53.568150 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.568063    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"]
Sep 09 19:08:53.572255 ip-10-0-134-63 kubenswrapper[2575]: W0909 19:08:53.572226    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe7e6540_6323_49bb_be3a_a49687822767.slice/crio-2bd27d4ec9e4105d65427a4dcca449d30b74c5c9d7bad094ba09b0321d00f885 WatchSource:0}: Error finding container 2bd27d4ec9e4105d65427a4dcca449d30b74c5c9d7bad094ba09b0321d00f885: Status 404 returned error can't find the container with id 2bd27d4ec9e4105d65427a4dcca449d30b74c5c9d7bad094ba09b0321d00f885
Sep 09 19:08:53.846243 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.846145    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns" event={"ID":"fe7e6540-6323-49bb-be3a-a49687822767","Type":"ContainerStarted","Data":"68cff0aa9a80ac19e70d22d6efb41bd1ec331baec11dbc42f4f42e6859de1c10"}
Sep 09 19:08:53.846243 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.846193    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns" event={"ID":"fe7e6540-6323-49bb-be3a-a49687822767","Type":"ContainerStarted","Data":"2bd27d4ec9e4105d65427a4dcca449d30b74c5c9d7bad094ba09b0321d00f885"}
Sep 09 19:08:53.869598 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:08:53.869541    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns" podStartSLOduration=1.869520869 podStartE2EDuration="1.869520869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:08:53.867724872 +0000 UTC m=+2222.105094665" watchObservedRunningTime="2026-09-09 19:08:53.869520869 +0000 UTC m=+2222.106890680"
Sep 09 19:09:03.265716 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:03.265676    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:09:03.266243 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:03.265737    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:09:03.278693 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:03.278664    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:09:03.912740 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:03.912707    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:09:10.273456 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:10.273414    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"]
Sep 09 19:09:10.881541 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:10.881488    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"]
Sep 09 19:09:25.295019 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.294946    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj" podUID="f2cb5b9a-275a-48e8-89db-c7eafd258c30" containerName="main" containerID="cri-o://45bb48404538a287792d67d9a9833cd53a3196ead14e08d71020989d773b494c" gracePeriod=45
Sep 09 19:09:25.554706 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.554628    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:09:25.577531 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.577462    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-tmp-dir\") pod \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") "
Sep 09 19:09:25.577720 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.577561    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-dshm\") pod \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") "
Sep 09 19:09:25.577720 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.577619    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-home\") pod \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") "
Sep 09 19:09:25.577720 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.577667    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hfxs\" (UniqueName: \"kubernetes.io/projected/f2cb5b9a-275a-48e8-89db-c7eafd258c30-kube-api-access-7hfxs\") pod \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") "
Sep 09 19:09:25.577891 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.577740    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cb5b9a-275a-48e8-89db-c7eafd258c30-tls-certs\") pod \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") "
Sep 09 19:09:25.577891 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.577772    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-model-cache\") pod \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\" (UID: \"f2cb5b9a-275a-48e8-89db-c7eafd258c30\") "
Sep 09 19:09:25.577995 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.577949    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-home" (OuterVolumeSpecName: "home") pod "f2cb5b9a-275a-48e8-89db-c7eafd258c30" (UID: "f2cb5b9a-275a-48e8-89db-c7eafd258c30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:09:25.578172 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.578127    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:09:25.578172 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.578156    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2cb5b9a-275a-48e8-89db-c7eafd258c30" (UID: "f2cb5b9a-275a-48e8-89db-c7eafd258c30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:09:25.578369 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.578250    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2cb5b9a-275a-48e8-89db-c7eafd258c30" (UID: "f2cb5b9a-275a-48e8-89db-c7eafd258c30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:09:25.580296 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.580259    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-dshm" (OuterVolumeSpecName: "dshm") pod "f2cb5b9a-275a-48e8-89db-c7eafd258c30" (UID: "f2cb5b9a-275a-48e8-89db-c7eafd258c30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:09:25.580440 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.580420    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2cb5b9a-275a-48e8-89db-c7eafd258c30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2cb5b9a-275a-48e8-89db-c7eafd258c30" (UID: "f2cb5b9a-275a-48e8-89db-c7eafd258c30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:09:25.580597 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.580569    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cb5b9a-275a-48e8-89db-c7eafd258c30-kube-api-access-7hfxs" (OuterVolumeSpecName: "kube-api-access-7hfxs") pod "f2cb5b9a-275a-48e8-89db-c7eafd258c30" (UID: "f2cb5b9a-275a-48e8-89db-c7eafd258c30"). InnerVolumeSpecName "kube-api-access-7hfxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:09:25.679314 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.679272    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2cb5b9a-275a-48e8-89db-c7eafd258c30-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:09:25.679314 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.679307    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:09:25.679575 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.679346    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:09:25.679575 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.679356    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2cb5b9a-275a-48e8-89db-c7eafd258c30-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:09:25.679575 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.679365    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hfxs\" (UniqueName: \"kubernetes.io/projected/f2cb5b9a-275a-48e8-89db-c7eafd258c30-kube-api-access-7hfxs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:09:25.903021 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:25.902951    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns" podUID="fe7e6540-6323-49bb-be3a-a49687822767" containerName="main" containerID="cri-o://68cff0aa9a80ac19e70d22d6efb41bd1ec331baec11dbc42f4f42e6859de1c10" gracePeriod=45
Sep 09 19:09:25.952176 ip-10-0-134-63 kubenswrapper[2575]: E0909 19:09:25.952136    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe7e6540_6323_49bb_be3a_a49687822767.slice/crio-conmon-68cff0aa9a80ac19e70d22d6efb41bd1ec331baec11dbc42f4f42e6859de1c10.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 19:09:26.019994 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.019959    2575 generic.go:356] "Generic (PLEG): container finished" podID="f2cb5b9a-275a-48e8-89db-c7eafd258c30" containerID="45bb48404538a287792d67d9a9833cd53a3196ead14e08d71020989d773b494c" exitCode=0
Sep 09 19:09:26.020168 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.020007    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"
Sep 09 19:09:26.020168 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.020036    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj" event={"ID":"f2cb5b9a-275a-48e8-89db-c7eafd258c30","Type":"ContainerDied","Data":"45bb48404538a287792d67d9a9833cd53a3196ead14e08d71020989d773b494c"}
Sep 09 19:09:26.020168 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.020076    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj" event={"ID":"f2cb5b9a-275a-48e8-89db-c7eafd258c30","Type":"ContainerDied","Data":"3d2b3496d73eb3916e68399097494e599910a956bd6c4cacdeeab471f2cded70"}
Sep 09 19:09:26.020168 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.020092    2575 scope.go:117] "RemoveContainer" containerID="45bb48404538a287792d67d9a9833cd53a3196ead14e08d71020989d773b494c"
Sep 09 19:09:26.031983 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.030485    2575 scope.go:117] "RemoveContainer" containerID="45bb48404538a287792d67d9a9833cd53a3196ead14e08d71020989d773b494c"
Sep 09 19:09:26.031983 ip-10-0-134-63 kubenswrapper[2575]: E0909 19:09:26.030956    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45bb48404538a287792d67d9a9833cd53a3196ead14e08d71020989d773b494c\": container with ID starting with 45bb48404538a287792d67d9a9833cd53a3196ead14e08d71020989d773b494c not found: ID does not exist" containerID="45bb48404538a287792d67d9a9833cd53a3196ead14e08d71020989d773b494c"
Sep 09 19:09:26.031983 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.030997    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45bb48404538a287792d67d9a9833cd53a3196ead14e08d71020989d773b494c"} err="failed to get container status \"45bb48404538a287792d67d9a9833cd53a3196ead14e08d71020989d773b494c\": rpc error: code = NotFound desc = could not find container \"45bb48404538a287792d67d9a9833cd53a3196ead14e08d71020989d773b494c\": container with ID starting with 45bb48404538a287792d67d9a9833cd53a3196ead14e08d71020989d773b494c not found: ID does not exist"
Sep 09 19:09:26.050641 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.050600    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"]
Sep 09 19:09:26.061305 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.061267    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9cde296a-kserve-6f56656b99-8xdxj"]
Sep 09 19:09:26.147429 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.147399    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:09:26.183425 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.183329    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-tmp-dir\") pod \"fe7e6540-6323-49bb-be3a-a49687822767\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") "
Sep 09 19:09:26.183425 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.183372    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-model-cache\") pod \"fe7e6540-6323-49bb-be3a-a49687822767\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") "
Sep 09 19:09:26.183691 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.183435    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7e6540-6323-49bb-be3a-a49687822767-tls-certs\") pod \"fe7e6540-6323-49bb-be3a-a49687822767\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") "
Sep 09 19:09:26.183691 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.183458    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-home\") pod \"fe7e6540-6323-49bb-be3a-a49687822767\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") "
Sep 09 19:09:26.183691 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.183529    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-dshm\") pod \"fe7e6540-6323-49bb-be3a-a49687822767\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") "
Sep 09 19:09:26.183691 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.183595    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6m2w\" (UniqueName: \"kubernetes.io/projected/fe7e6540-6323-49bb-be3a-a49687822767-kube-api-access-x6m2w\") pod \"fe7e6540-6323-49bb-be3a-a49687822767\" (UID: \"fe7e6540-6323-49bb-be3a-a49687822767\") "
Sep 09 19:09:26.183888 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.183662    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe7e6540-6323-49bb-be3a-a49687822767" (UID: "fe7e6540-6323-49bb-be3a-a49687822767"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:09:26.183888 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.183757    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe7e6540-6323-49bb-be3a-a49687822767" (UID: "fe7e6540-6323-49bb-be3a-a49687822767"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:09:26.183888 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.183786    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-home" (OuterVolumeSpecName: "home") pod "fe7e6540-6323-49bb-be3a-a49687822767" (UID: "fe7e6540-6323-49bb-be3a-a49687822767"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:09:26.184035 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.183980    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-tmp-dir\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:09:26.184035 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.184001    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-model-cache\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:09:26.184035 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.184013    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-home\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:09:26.185812 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.185782    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-dshm" (OuterVolumeSpecName: "dshm") pod "fe7e6540-6323-49bb-be3a-a49687822767" (UID: "fe7e6540-6323-49bb-be3a-a49687822767"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:09:26.185940 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.185908    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe7e6540-6323-49bb-be3a-a49687822767-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe7e6540-6323-49bb-be3a-a49687822767" (UID: "fe7e6540-6323-49bb-be3a-a49687822767"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:09:26.185940 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.185924    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7e6540-6323-49bb-be3a-a49687822767-kube-api-access-x6m2w" (OuterVolumeSpecName: "kube-api-access-x6m2w") pod "fe7e6540-6323-49bb-be3a-a49687822767" (UID: "fe7e6540-6323-49bb-be3a-a49687822767"). InnerVolumeSpecName "kube-api-access-x6m2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:09:26.284994 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.284947    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6m2w\" (UniqueName: \"kubernetes.io/projected/fe7e6540-6323-49bb-be3a-a49687822767-kube-api-access-x6m2w\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:09:26.284994 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.284987    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7e6540-6323-49bb-be3a-a49687822767-tls-certs\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:09:26.284994 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.284999    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7e6540-6323-49bb-be3a-a49687822767-dshm\") on node \"ip-10-0-134-63.ec2.internal\" DevicePath \"\""
Sep 09 19:09:26.321417 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:26.321381    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2cb5b9a-275a-48e8-89db-c7eafd258c30" path="/var/lib/kubelet/pods/f2cb5b9a-275a-48e8-89db-c7eafd258c30/volumes"
Sep 09 19:09:27.027232 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:27.027198    2575 generic.go:356] "Generic (PLEG): container finished" podID="fe7e6540-6323-49bb-be3a-a49687822767" containerID="68cff0aa9a80ac19e70d22d6efb41bd1ec331baec11dbc42f4f42e6859de1c10" exitCode=0
Sep 09 19:09:27.027232 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:27.027236    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns" event={"ID":"fe7e6540-6323-49bb-be3a-a49687822767","Type":"ContainerDied","Data":"68cff0aa9a80ac19e70d22d6efb41bd1ec331baec11dbc42f4f42e6859de1c10"}
Sep 09 19:09:27.027478 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:27.027264    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns" event={"ID":"fe7e6540-6323-49bb-be3a-a49687822767","Type":"ContainerDied","Data":"2bd27d4ec9e4105d65427a4dcca449d30b74c5c9d7bad094ba09b0321d00f885"}
Sep 09 19:09:27.027478 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:27.027282    2575 scope.go:117] "RemoveContainer" containerID="68cff0aa9a80ac19e70d22d6efb41bd1ec331baec11dbc42f4f42e6859de1c10"
Sep 09 19:09:27.027478 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:27.027240    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"
Sep 09 19:09:27.036468 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:27.036446    2575 scope.go:117] "RemoveContainer" containerID="68cff0aa9a80ac19e70d22d6efb41bd1ec331baec11dbc42f4f42e6859de1c10"
Sep 09 19:09:27.036827 ip-10-0-134-63 kubenswrapper[2575]: E0909 19:09:27.036807    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cff0aa9a80ac19e70d22d6efb41bd1ec331baec11dbc42f4f42e6859de1c10\": container with ID starting with 68cff0aa9a80ac19e70d22d6efb41bd1ec331baec11dbc42f4f42e6859de1c10 not found: ID does not exist" containerID="68cff0aa9a80ac19e70d22d6efb41bd1ec331baec11dbc42f4f42e6859de1c10"
Sep 09 19:09:27.036902 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:27.036835    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cff0aa9a80ac19e70d22d6efb41bd1ec331baec11dbc42f4f42e6859de1c10"} err="failed to get container status \"68cff0aa9a80ac19e70d22d6efb41bd1ec331baec11dbc42f4f42e6859de1c10\": rpc error: code = NotFound desc = could not find container \"68cff0aa9a80ac19e70d22d6efb41bd1ec331baec11dbc42f4f42e6859de1c10\": container with ID starting with 68cff0aa9a80ac19e70d22d6efb41bd1ec331baec11dbc42f4f42e6859de1c10 not found: ID does not exist"
Sep 09 19:09:27.048299 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:27.048264    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"]
Sep 09 19:09:27.054652 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:27.054617    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-9cde296a-kserve-69f9f989f9-w5pns"]
Sep 09 19:09:28.320927 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:28.320888    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe7e6540-6323-49bb-be3a-a49687822767" path="/var/lib/kubelet/pods/fe7e6540-6323-49bb-be3a-a49687822767/volumes"
Sep 09 19:09:52.792042 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:09:52.792004    2575 scope.go:117] "RemoveContainer" containerID="e87c1ff0d4331f0c008e89656b385e539d9a5a7e28b257146f5dbdf232a45f4e"
Sep 09 19:10:00.251175 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:00.251138    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f6fc7bf65-lpkhr_b0746fc4-4d76-49f3-9d34-627a0fd2cbaf/authorino/0.log"
Sep 09 19:10:06.671480 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:06.671447    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f66f47469-nnn72_1fd60a9f-60c0-4b72-800f-e117c90283c8/copy-manifests/0.log"
Sep 09 19:10:06.693120 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:06.693089    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f66f47469-nnn72_1fd60a9f-60c0-4b72-800f-e117c90283c8/manager/0.log"
Sep 09 19:10:08.397307 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:08.397273    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f6fc7bf65-lpkhr_b0746fc4-4d76-49f3-9d34-627a0fd2cbaf/authorino/0.log"
Sep 09 19:10:09.130688 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:09.130653    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-djd9z_d5d8952a-3efc-444c-b2fb-b41ee743bc67/manager/0.log"
Sep 09 19:10:09.884222 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:09.884190    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799dfc8bc6-flvvx_3f6f112a-49fb-4edf-af39-8603398fab38/kube-auth-proxy/0.log"
Sep 09 19:10:10.294323 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:10.294237    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-b4lhf_69804e5e-c502-42ae-aee8-d4e2f7ea2262/payload-processing/0.log"
Sep 09 19:10:10.789118 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:10.789081    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-8tw78_a9d584a4-1535-4fde-8387-850b2754b8c8/main/0.log"
Sep 09 19:10:11.255455 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:11.255421    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f9vs8j_2184298b-d8f3-4c77-a654-7c383d714aff/main/0.log"
Sep 09 19:10:11.508210 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:11.508133    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-rjjtz_d1e4710a-d531-4339-a84c-4e00d878523e/main/0.log"
Sep 09 19:10:24.070722 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:24.070683    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7vhcs_3526bb20-1563-4c64-970b-e159d015bd3d/global-pull-secret-syncer/0.log"
Sep 09 19:10:24.334275 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:24.334193    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jnb9f_c58852df-cacb-4054-8f99-18344f10936d/konnectivity-agent/0.log"
Sep 09 19:10:24.406665 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:24.406637    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-63.ec2.internal_b14058b333631407a76dac7ca4cc84a1/haproxy/0.log"
Sep 09 19:10:28.936367 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:28.936319    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-f6fc7bf65-lpkhr_b0746fc4-4d76-49f3-9d34-627a0fd2cbaf/authorino/0.log"
Sep 09 19:10:29.238996 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:29.238956    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-djd9z_d5d8952a-3efc-444c-b2fb-b41ee743bc67/manager/0.log"
Sep 09 19:10:30.610665 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:30.610631    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4eea30ae-084d-4eb7-9b57-08d350bab832/alertmanager/0.log"
Sep 09 19:10:30.643312 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:30.643274    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4eea30ae-084d-4eb7-9b57-08d350bab832/config-reloader/0.log"
Sep 09 19:10:30.671131 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:30.671092    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4eea30ae-084d-4eb7-9b57-08d350bab832/kube-rbac-proxy-web/0.log"
Sep 09 19:10:30.699676 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:30.699645    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4eea30ae-084d-4eb7-9b57-08d350bab832/kube-rbac-proxy/0.log"
Sep 09 19:10:30.736710 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:30.736681    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4eea30ae-084d-4eb7-9b57-08d350bab832/kube-rbac-proxy-metric/0.log"
Sep 09 19:10:30.782625 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:30.782581    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4eea30ae-084d-4eb7-9b57-08d350bab832/prom-label-proxy/0.log"
Sep 09 19:10:30.822191 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:30.822160    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4eea30ae-084d-4eb7-9b57-08d350bab832/init-config-reloader/0.log"
Sep 09 19:10:30.870938 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:30.870856    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-7jqpl_b9722a7d-f4e8-49ba-aa95-a0adaad2f4a8/cluster-monitoring-operator/0.log"
Sep 09 19:10:31.007130 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.007098    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-64549dc5b-hczq5_78cad44e-5dd0-41be-9846-367f23eeb558/metrics-server/0.log"
Sep 09 19:10:31.089876 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.089842    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4c5fq_3c9db26d-4df7-48e0-9a7c-eb664a8eeb53/node-exporter/0.log"
Sep 09 19:10:31.124041 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.123961    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4c5fq_3c9db26d-4df7-48e0-9a7c-eb664a8eeb53/kube-rbac-proxy/0.log"
Sep 09 19:10:31.156948 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.156917    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4c5fq_3c9db26d-4df7-48e0-9a7c-eb664a8eeb53/init-textfile/0.log"
Sep 09 19:10:31.384395 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.384309    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-n6pjv_f9f44554-99ce-4fdf-9b59-7635b1cc48e7/kube-rbac-proxy-main/0.log"
Sep 09 19:10:31.413034 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.412998    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-n6pjv_f9f44554-99ce-4fdf-9b59-7635b1cc48e7/kube-rbac-proxy-self/0.log"
Sep 09 19:10:31.448006 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.447975    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-n6pjv_f9f44554-99ce-4fdf-9b59-7635b1cc48e7/openshift-state-metrics/0.log"
Sep 09 19:10:31.508254 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.508229    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b492f3c-9415-4ea1-854c-ca896f244e49/prometheus/0.log"
Sep 09 19:10:31.558383 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.558354    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b492f3c-9415-4ea1-854c-ca896f244e49/config-reloader/0.log"
Sep 09 19:10:31.608830 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.608803    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b492f3c-9415-4ea1-854c-ca896f244e49/thanos-sidecar/0.log"
Sep 09 19:10:31.640847 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.640766    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b492f3c-9415-4ea1-854c-ca896f244e49/kube-rbac-proxy-web/0.log"
Sep 09 19:10:31.696938 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.696911    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b492f3c-9415-4ea1-854c-ca896f244e49/kube-rbac-proxy/0.log"
Sep 09 19:10:31.732092 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.732066    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b492f3c-9415-4ea1-854c-ca896f244e49/kube-rbac-proxy-thanos/0.log"
Sep 09 19:10:31.766488 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.766459    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b492f3c-9415-4ea1-854c-ca896f244e49/init-config-reloader/0.log"
Sep 09 19:10:31.887439 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.887412    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-dmhfm_9e88beb3-f59b-4503-8215-a3e8f4cde9c0/prometheus-operator-admission-webhook/0.log"
Sep 09 19:10:31.929046 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.928951    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5876879d86-94gxt_ef8d6649-c9cc-4933-b538-4041cd3f5bec/telemeter-client/0.log"
Sep 09 19:10:31.971770 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:31.971735    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5876879d86-94gxt_ef8d6649-c9cc-4933-b538-4041cd3f5bec/reload/0.log"
Sep 09 19:10:32.005886 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:32.005853    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5876879d86-94gxt_ef8d6649-c9cc-4933-b538-4041cd3f5bec/kube-rbac-proxy/0.log"
Sep 09 19:10:34.186741 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.186700    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"]
Sep 09 19:10:34.187149 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.187103    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe7e6540-6323-49bb-be3a-a49687822767" containerName="main"
Sep 09 19:10:34.187149 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.187117    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe7e6540-6323-49bb-be3a-a49687822767" containerName="main"
Sep 09 19:10:34.187149 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.187133    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2cb5b9a-275a-48e8-89db-c7eafd258c30" containerName="main"
Sep 09 19:10:34.187149 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.187139    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2cb5b9a-275a-48e8-89db-c7eafd258c30" containerName="main"
Sep 09 19:10:34.187285 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.187205    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2cb5b9a-275a-48e8-89db-c7eafd258c30" containerName="main"
Sep 09 19:10:34.187285 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.187214    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe7e6540-6323-49bb-be3a-a49687822767" containerName="main"
Sep 09 19:10:34.190487 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.190466    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.195786 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.195757    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dz2cr\"/\"default-dockercfg-dnjv8\""
Sep 09 19:10:34.195946 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.195766    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dz2cr\"/\"openshift-service-ca.crt\""
Sep 09 19:10:34.195946 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.195768    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dz2cr\"/\"kube-root-ca.crt\""
Sep 09 19:10:34.209060 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.209028    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"]
Sep 09 19:10:34.270911 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.270869    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-sys\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.271114 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.270928    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27pwg\" (UniqueName: \"kubernetes.io/projected/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-kube-api-access-27pwg\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.271114 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.270995    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-proc\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.271114 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.271100    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-podres\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.271235 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.271181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-lib-modules\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.372377 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.372342    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-podres\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.372632 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.372587    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-podres\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.372859 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.372824    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-lib-modules\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.373000 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.372905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-sys\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.373000 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.372984    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27pwg\" (UniqueName: \"kubernetes.io/projected/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-kube-api-access-27pwg\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.373129 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.372995    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-lib-modules\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.373129 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.373068    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-proc\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.373241 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.373203    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-proc\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.373241 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.373228    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-sys\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.383121 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.383087    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27pwg\" (UniqueName: \"kubernetes.io/projected/59bf7e05-e271-4e4b-a5f2-d9983751d7dc-kube-api-access-27pwg\") pod \"perf-node-gather-daemonset-cdppr\" (UID: \"59bf7e05-e271-4e4b-a5f2-d9983751d7dc\") " pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.405087 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.405054    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/2.log"
Sep 09 19:10:34.409298 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.409276    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-nmt4x_b2848f11-65c4-4624-8cae-a9c7de831abf/console-operator/3.log"
Sep 09 19:10:34.500428 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.500326    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:34.798427 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:34.798398    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"]
Sep 09 19:10:34.800329 ip-10-0-134-63 kubenswrapper[2575]: W0909 19:10:34.800300    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod59bf7e05_e271_4e4b_a5f2_d9983751d7dc.slice/crio-22c3100b6de930f0aae79b6a372bbbf245ef32a43314843659ee4ae7913ebbdd WatchSource:0}: Error finding container 22c3100b6de930f0aae79b6a372bbbf245ef32a43314843659ee4ae7913ebbdd: Status 404 returned error can't find the container with id 22c3100b6de930f0aae79b6a372bbbf245ef32a43314843659ee4ae7913ebbdd
Sep 09 19:10:35.383253 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:35.383210    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr" event={"ID":"59bf7e05-e271-4e4b-a5f2-d9983751d7dc","Type":"ContainerStarted","Data":"1d98d7c64888d6fd53ae181244cccd537906cb01e30790625d0db5dc47f04d57"}
Sep 09 19:10:35.383704 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:35.383279    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr" event={"ID":"59bf7e05-e271-4e4b-a5f2-d9983751d7dc","Type":"ContainerStarted","Data":"22c3100b6de930f0aae79b6a372bbbf245ef32a43314843659ee4ae7913ebbdd"}
Sep 09 19:10:35.383704 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:35.383308    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:35.404742 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:35.404679    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr" podStartSLOduration=1.404661811 podStartE2EDuration="1.404661811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:10:35.4037869 +0000 UTC m=+2323.641156716" watchObservedRunningTime="2026-09-09 19:10:35.404661811 +0000 UTC m=+2323.642031627"
Sep 09 19:10:35.811213 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:35.811132    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-9ftgh_87708efc-02ee-4a2a-88c6-f519a08db330/volume-data-source-validator/0.log"
Sep 09 19:10:37.097617 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:37.097591    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lpmq9_4c571b1e-36e0-4647-a241-511ce5cfa737/dns/0.log"
Sep 09 19:10:37.206664 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:37.206629    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lpmq9_4c571b1e-36e0-4647-a241-511ce5cfa737/kube-rbac-proxy/0.log"
Sep 09 19:10:37.377999 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:37.377974    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lpvdl_d4099f20-afa3-482b-a22e-c4d59554f7fd/dns-node-resolver/0.log"
Sep 09 19:10:37.976798 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:37.976761    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6b7bd5bbf4-6sctb_7b27f60e-eb9e-41f5-8141-6c2ed0448395/registry/0.log"
Sep 09 19:10:37.984055 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:37.984023    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6b7bd5bbf4-6sctb_7b27f60e-eb9e-41f5-8141-6c2ed0448395/registry/1.log"
Sep 09 19:10:38.076199 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:38.076169    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t75rf_1753a297-ac2b-49cd-b9c9-459465d3d912/node-ca/0.log"
Sep 09 19:10:39.542493 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:39.542463    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799dfc8bc6-flvvx_3f6f112a-49fb-4edf-af39-8603398fab38/kube-auth-proxy/0.log"
Sep 09 19:10:39.696725 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:39.696689    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-b4lhf_69804e5e-c502-42ae-aee8-d4e2f7ea2262/payload-processing/0.log"
Sep 09 19:10:40.735728 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:40.735697    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xsjjs_d7b9ed88-9b45-457d-9e8e-9863f49d267b/serve-healthcheck-canary/0.log"
Sep 09 19:10:41.314397 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:41.314350    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-vz6kz_f82752ac-77f8-44e5-a619-d1e1008923ce/insights-operator/0.log"
Sep 09 19:10:41.318009 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:41.317978    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-vz6kz_f82752ac-77f8-44e5-a619-d1e1008923ce/insights-operator/1.log"
Sep 09 19:10:41.399699 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:41.399659    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dz2cr/perf-node-gather-daemonset-cdppr"
Sep 09 19:10:41.556414 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:41.556389    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tlfq9_dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3/kube-rbac-proxy/0.log"
Sep 09 19:10:41.585680 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:41.585598    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tlfq9_dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3/exporter/0.log"
Sep 09 19:10:41.615120 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:41.615089    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tlfq9_dc0bd9c1-3dd5-4b8b-b581-844fa31f94e3/extractor/0.log"
Sep 09 19:10:45.006018 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:45.005990    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799dfc8bc6-flvvx_3f6f112a-49fb-4edf-af39-8603398fab38/kube-auth-proxy/0.log"
Sep 09 19:10:45.114431 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:45.114392    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f66f47469-nnn72_1fd60a9f-60c0-4b72-800f-e117c90283c8/manager/0.log"
Sep 09 19:10:45.120140 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:45.120108    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-b4lhf_69804e5e-c502-42ae-aee8-d4e2f7ea2262/payload-processing/0.log"
Sep 09 19:10:45.140696 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:45.140665    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f66f47469-nnn72_1fd60a9f-60c0-4b72-800f-e117c90283c8/copy-manifests/0.log"
Sep 09 19:10:46.691333 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:46.691296    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-96799d79b-sfldg_a1732ed2-a6c1-443b-8fae-92cc9b4542ea/manager/0.log"
Sep 09 19:10:47.384789 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:47.384759    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-799dfc8bc6-flvvx_3f6f112a-49fb-4edf-af39-8603398fab38/kube-auth-proxy/0.log"
Sep 09 19:10:47.487331 ip-10-0-134-63 kubenswrapper[2575]: I0909 19:10:47.487291    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-b4lhf_69804e5e-c502-42ae-aee8-d4e2f7ea2262/payload-processing/0.log"