Sep 10 22:59:35.309551 ip-10-0-130-246 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 22:59:35.309693 ip-10-0-130-246 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 22:59:35.309711 ip-10-0-130-246 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 22:59:35.310177 ip-10-0-130-246 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 22:59:44.728470 ip-10-0-130-246 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 10 22:59:44.728484 ip-10-0-130-246 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a29d6ddcdc0a4d27ac87326e0ba09616 --
Sep 10 23:01:19.201801 ip-10-0-130-246 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 23:01:19.614421 ip-10-0-130-246 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:01:19.614421 ip-10-0-130-246 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 23:01:19.614421 ip-10-0-130-246 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:01:19.614421 ip-10-0-130-246 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 23:01:19.614421 ip-10-0-130-246 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 23:01:19.617395 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.617309    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 23:01:19.624640 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624615    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:19.624640 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624635    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:19.624640 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624639    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:19.624640 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624642    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:19.624640 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624645    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624649    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624652    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624655    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624657    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624662    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624667    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624669    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624672    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624675    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624678    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624680    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624683    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624686    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624689    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624691    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624694    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624696    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624699    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624702    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:19.624855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624705    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624707    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624710    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624713    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624716    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624718    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624721    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624725    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624733    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624735    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624738    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624740    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624743    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624746    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624750    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624752    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624755    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624760    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624762    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:19.625338 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624766    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624770    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624774    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624777    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624780    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624783    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624786    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624789    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624792    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624795    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624798    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624801    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624804    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624807    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624809    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624812    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624814    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624817    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624820    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624822    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:19.625915 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624825    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624827    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624830    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624832    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624835    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624837    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624840    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624844    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624847    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624850    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624854    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624857    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624859    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624862    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624865    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624868    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624871    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624874    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624876    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:19.626407 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624880    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624886    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624889    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624892    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624895    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624897    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624900    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624903    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624905    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624908    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624911    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624913    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624916    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624919    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624921    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624924    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.624927    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625682    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625690    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625693    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:19.626893 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625698    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625702    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625705    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625708    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625711    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625713    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625716    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625719    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625721    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625724    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625727    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625729    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625732    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625734    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625738    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625740    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625750    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625753    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625756    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625759    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:19.627379 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625762    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625764    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625767    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625770    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625772    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625775    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625777    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625780    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625782    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625785    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625787    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625790    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625793    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625796    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625801    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625804    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625807    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625810    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625813    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625816    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:19.627931 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625819    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625821    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625824    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625826    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625830    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625832    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625835    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625838    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625841    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625844    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625846    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625849    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625851    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625854    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625858    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625861    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625864    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625867    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625869    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:19.628421 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625872    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625874    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625877    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625880    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625884    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625888    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625890    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625893    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625896    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625898    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625901    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625903    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625906    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625908    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625910    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625913    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625916    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625919    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625924    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:19.628895 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625926    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625929    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625932    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625935    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625937    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625940    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625942    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625945    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625947    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625949    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625952    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625954    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625957    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625960    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625962    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625965    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625967    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.625969    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626352    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626357    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:19.629361 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626360    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626363    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626365    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626369    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626371    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626374    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626376    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626379    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626382    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626384    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626387    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626389    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626392    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626394    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626397    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626400    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626403    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626405    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626408    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626411    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:19.629868 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626413    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626416    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626419    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626422    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626425    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626427    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626430    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626433    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626435    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626438    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626443    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626446    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626449    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626451    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626454    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626462    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626465    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626467    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626470    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626472    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:19.630353 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626475    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626477    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626480    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626482    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626485    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626487    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626504    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626507    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626510    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626513    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626515    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626518    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626520    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626523    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626525    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626528    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626531    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626533    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626536    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626539    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:19.630840 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626542    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626545    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626551    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626554    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626558    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626561    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626565    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626568    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626570    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626573    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626575    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626578    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626580    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626583    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626586    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626589    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626592    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626596    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626598    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:19.631325 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626601    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626605    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626608    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626610    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626613    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626617    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626620    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626623    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626626    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626628    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626631    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626633    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626636    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626639    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626641    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626645    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626648    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.626651    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626723    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626755    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 23:01:19.631796 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626762    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626771    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626775    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626779    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626783    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626787    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626790    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626793    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626797    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626800    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626803    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626806    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626809    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626812    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626817    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626820    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626824    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626826    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626830    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626833    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626836    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626840    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626843    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 23:01:19.632271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626846    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626851    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626854    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626856    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626860    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626863    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626867    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626874    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626877    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626880    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626882    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626886    2563 flags.go:64] FLAG: --enable-server="true"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626889    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626894    2563 flags.go:64] FLAG: --event-burst="100"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626897    2563 flags.go:64] FLAG: --event-qps="50"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626900    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626904    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626907    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626911    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626914    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626916    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626919    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626923    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626926    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626929    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 23:01:19.632874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626932    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626935    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626938    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626941    2563 flags.go:64] FLAG: --feature-gates=""
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626944    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626947    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626950    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626953    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626959    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626962    2563 flags.go:64] FLAG: --help="false"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626965    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-130-246.ec2.internal"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626969    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626972    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626975    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626978    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626981    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626984    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626987    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626990    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626992    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626995    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.626998    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627001    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627004    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 23:01:19.633479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627007    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627010    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627013    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627016    2563 flags.go:64] FLAG: --lock-file=""
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627018    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627021    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627024    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627029    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627033    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627035    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627038    2563 flags.go:64] FLAG: --logging-format="text"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627041    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627044    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627047    2563 flags.go:64] FLAG: --manifest-url=""
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627050    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627054    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627058    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627062    2563 flags.go:64] FLAG: --max-pods="110"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627066    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627069    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627072    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627075    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627079    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627081    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627084    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 23:01:19.634106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627092    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627095    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627098    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627101    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627104    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627109    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627112    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627115    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627118    2563 flags.go:64] FLAG: --port="10250"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627122    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627124    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09844d4fef602d44c"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627128    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627130    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627133    2563 flags.go:64] FLAG: --register-node="true"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627136    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627140    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627144    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627147    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627149    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627153    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627156    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627159    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627162    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627166    2563 flags.go:64] FLAG: --runonce="false"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627169    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 23:01:19.634769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627172    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627175    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627179    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627182    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627185    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627189    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627192    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627194    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627197    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627200    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627203    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627206    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627209    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627212    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627217    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627220    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627223    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627227    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627230    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627232    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627235    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627238    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627241    2563 flags.go:64] FLAG: --v="2"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627245    2563 flags.go:64] FLAG: --version="false"
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627250    2563 flags.go:64] FLAG: --vmodule=""
Sep 10 23:01:19.635409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627253    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.627256    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627368    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627372    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627376    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627380    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627383    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627386    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627389    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627394    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627396    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627400    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627403    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627405    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627408    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627411    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627414    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627417    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627420    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627423    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:19.636021 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627426    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627428    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627433    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627437    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627440    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627443    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627446    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627449    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627452    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627454    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627458    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627460    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627464    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627467    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627469    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627472    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627475    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627479    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627482    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:19.636567 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627484    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627487    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627504    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627507    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627510    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627513    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627516    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627519    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627522    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627525    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627528    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627531    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627534    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627536    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627539    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627542    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627545    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627547    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627550    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627553    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:19.637032 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627556    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627559    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627561    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627564    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627567    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627570    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627573    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627576    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627578    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627581    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627585    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627589    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627594    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627598    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627612    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627615    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627618    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627621    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627624    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:19.637533 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627627    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627630    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627633    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627636    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627638    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627641    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627644    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627647    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627650    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627652    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627655    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627658    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627661    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627663    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627666    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627669    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627671    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627674    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627677    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:19.638013 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627679    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:19.638474 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627683    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:19.638474 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627685    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:19.638474 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.627688    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:19.638474 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.628361    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:01:19.638474 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.636249    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 23:01:19.638474 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.636266    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 23:01:19.638474 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.637133    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 23:01:19.638474 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.637141    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 23:01:19.638474 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637198    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:19.638474 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637205    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:19.638474 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637209    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:19.638474 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637212    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:19.638474 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637215    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:19.638474 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637218    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637221    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637224    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637227    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637230    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637232    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637235    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637238    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637241    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637243    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637246    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637249    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637252    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637254    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637257    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637259    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637262    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637265    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637267    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637270    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:19.638848 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637272    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637275    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637279    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637283    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637288    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637291    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637294    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637297    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637300    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637302    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637305    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637308    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637310    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637313    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637315    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637318    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637320    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637323    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637326    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:19.639343 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637328    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637332    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637336    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637339    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637342    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637345    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637348    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637351    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637354    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637357    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637359    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637362    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637365    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637367    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637370    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637372    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637375    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637378    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637381    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637384    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:19.639811 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637387    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637389    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637392    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637395    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637397    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637400    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637402    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637405    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637407    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637410    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637412    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637415    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637417    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637420    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637422    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637425    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637427    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637430    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637433    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637435    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:19.640291 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637438    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:19.640794 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637441    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:19.640794 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637443    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:19.640794 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637446    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:19.640794 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637449    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:19.640794 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637451    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:19.640794 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637454    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:19.640794 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637456    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:19.640794 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637459    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:19.640794 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637462    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:19.640794 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637465    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:19.640794 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637468    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:19.640794 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637471    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:19.640794 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637473    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:19.640794 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637476    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:19.641127 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.637481    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:01:19.641127 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637613    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 23:01:19.641127 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637617    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 23:01:19.641127 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637621    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 23:01:19.641127 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637623    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 23:01:19.641127 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637631    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 23:01:19.641127 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637633    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 23:01:19.641127 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637636    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 23:01:19.641127 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637639    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 23:01:19.641127 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637643    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 23:01:19.641127 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637647    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 23:01:19.641127 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637649    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 23:01:19.641127 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637652    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 23:01:19.641127 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637655    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637657    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637660    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637663    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637665    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637668    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637670    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637673    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637676    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637678    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637681    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637683    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637686    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637689    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637692    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637695    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637699    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637703    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637706    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637709    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 23:01:19.641488 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637712    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637715    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637717    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637720    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637723    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637726    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637728    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637731    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637734    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637736    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637738    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637741    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637751    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637754    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637756    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637759    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637762    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637764    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637767    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637770    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 23:01:19.642024 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637772    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637775    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637779    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637783    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637786    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637789    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637792    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637795    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637798    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637802    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637805    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637808    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637811    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637813    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637816    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637818    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637821    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637823    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637826    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 23:01:19.642550 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637829    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637831    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637834    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637836    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637839    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637841    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637844    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637846    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637849    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637851    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637854    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637856    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637859    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637861    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637864    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637866    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637869    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637871    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637874    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637879    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 23:01:19.642996 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637882    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 23:01:19.643472 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637885    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 23:01:19.643472 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637887    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 23:01:19.643472 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637890    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 23:01:19.643472 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637892    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 23:01:19.643472 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637895    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 23:01:19.643472 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637897    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 23:01:19.643472 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:19.637900    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 23:01:19.643472 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.637905    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 23:01:19.643472 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.638010    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 23:01:19.643472 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.641292    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 23:01:19.643472 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.642418    2563 server.go:1019] "Starting client certificate rotation"
Sep 10 23:01:19.643472 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.642515    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 23:01:19.643472 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.642548    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 23:01:19.646179 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.646166    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 23:01:19.647297 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.647280    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 23:01:19.657717 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.657699    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 10 23:01:19.661185 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.661171    2563 log.go:25] "Validated CRI v1 image API"
Sep 10 23:01:19.664229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.664210    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 23:01:19.666811 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.666787    2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 90b6c209-fbb3-41e0-83ca-d94b3b78cca6:/dev/nvme0n1p3 ab38db3a-76a0-4513-bbfb-7971401d0643:/dev/nvme0n1p4]
Sep 10 23:01:19.666887 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.666809    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 23:01:19.671917 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.671790    2563 manager.go:216] Machine: {Timestamp:2026-09-10 23:01:19.670425073 +0000 UTC m=+0.362002267 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3124367 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a63c06d4dabf6f73df35a807be6ca SystemUUID:ec2a63c0-6d4d-abf6-f73d-f35a807be6ca BootID:a29d6ddc-dc0a-4d27-ac87-326e0ba09616 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6d:36:13:73:df Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6d:36:13:73:df Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:f2:34:c2:6c:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 23:01:19.671917 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.671907    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 23:01:19.672084 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.672018    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 23:01:19.673221 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.673206    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 23:01:19.675446 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.675415    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 23:01:19.675605 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.675448    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-246.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 23:01:19.677047 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.677037    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 23:01:19.677087 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.677050    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 23:01:19.677087 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.677058    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 23:01:19.677915 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.677905    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 23:01:19.679195 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.679185    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 23:01:19.683897 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.683884    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 23:01:19.684010 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.684000    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 23:01:19.684043 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.684032    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 23:01:19.684076 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.684045    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 23:01:19.684076 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.684070    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 23:01:19.684134 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.684084    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 10 23:01:19.684134 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.684114    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 23:01:19.684572 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.684553    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 23:01:19.684572 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.684574    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 23:01:19.687793 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.687777    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 23:01:19.690610 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.690590    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 23:01:19.690666 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.690628    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 23:01:19.692806 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.692793    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 23:01:19.692866 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.692811    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 23:01:19.692866 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.692817    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 23:01:19.692866 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.692823    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 23:01:19.692866 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.692838    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 23:01:19.692866 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.692846    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 23:01:19.692866 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.692861    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 23:01:19.692866 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.692867    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 23:01:19.693054 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.692874    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 23:01:19.693054 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.692879    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 23:01:19.693054 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.692888    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 23:01:19.693054 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.692894    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 23:01:19.693839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.693828    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 23:01:19.693877 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.693840    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 23:01:19.694223 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.694200    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-246.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 23:01:19.694268 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.694201    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 23:01:19.697524 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.697509    2563 server.go:1268] "Started kubelet"
Sep 10 23:01:19.697653 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.697572    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 23:01:19.697653 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.697598    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 23:01:19.697762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.697666    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 23:01:19.698255 ip-10-0-130-246 systemd[1]: Started Kubernetes Kubelet.
Sep 10 23:01:19.698993 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.698971    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 23:01:19.704038 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.703659    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 23:01:19.707244 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.707219    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 23:01:19.709460 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.709438    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 23:01:19.709988 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.709969    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-246.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 23:01:19.710192 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.710177    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 23:01:19.710285 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.710241    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-246.ec2.internal\" not found"
Sep 10 23:01:19.710862 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.710846    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 23:01:19.710932 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.710866    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 23:01:19.710932 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.710845    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 23:01:19.710932 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.710917    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 23:01:19.711059 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.710935    2563 factory.go:55] Registering systemd factory
Sep 10 23:01:19.711059 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.710943    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 10 23:01:19.711059 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.710964    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 23:01:19.711059 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.710971    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 23:01:19.711226 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.710032    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-246.ec2.internal.18d417f1e6ab224b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-246.ec2.internal,UID:ip-10-0-130-246.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-246.ec2.internal,},FirstTimestamp:2026-09-10 23:01:19.697461835 +0000 UTC m=+0.389039010,LastTimestamp:2026-09-10 23:01:19.697461835 +0000 UTC m=+0.389039010,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-246.ec2.internal,}"
Sep 10 23:01:19.711226 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.711145    2563 factory.go:153] Registering CRI-O factory
Sep 10 23:01:19.711226 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.711161    2563 factory.go:223] Registration of the crio container factory successfully
Sep 10 23:01:19.711226 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.711183    2563 factory.go:103] Registering Raw factory
Sep 10 23:01:19.711226 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.711197    2563 manager.go:1196] Started watching for new ooms in manager
Sep 10 23:01:19.712289 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.712274    2563 manager.go:349] Starting recovery of all containers
Sep 10 23:01:19.716344 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.716321    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-246.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 23:01:19.716474 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.716456    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 23:01:19.722190 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.722073    2563 manager.go:354] Recovery completed
Sep 10 23:01:19.725058 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.725034    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f44rr"
Sep 10 23:01:19.726859 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.726846    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:19.728922 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.728907    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:19.728988 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.728937    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:19.728988 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.728948    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:19.729471 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.729457    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 23:01:19.729471 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.729469    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 23:01:19.729604 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.729478    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:19.729604 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.729487    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 23:01:19.730219 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.730206    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:19.730278 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.730230    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:19.730278 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.730240    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:19.731641 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.731626    2563 policy_none.go:49] "None policy: Start"
Sep 10 23:01:19.731719 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.731644    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 23:01:19.731719 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.731658    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 23:01:19.731832 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.731699    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-246.ec2.internal.18d417f1e88b2f88  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-246.ec2.internal,UID:ip-10-0-130-246.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-246.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-246.ec2.internal,},FirstTimestamp:2026-09-10 23:01:19.728922504 +0000 UTC m=+0.420499678,LastTimestamp:2026-09-10 23:01:19.728922504 +0000 UTC m=+0.420499678,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-246.ec2.internal,}"
Sep 10 23:01:19.732303 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.732284    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f44rr"
Sep 10 23:01:19.733701 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.733686    2563 policy_none.go:47] "Start"
Sep 10 23:01:19.762098 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.762083    2563 manager.go:337] "Starting Device Plugin manager"
Sep 10 23:01:19.785649 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.762119    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 23:01:19.785649 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.762131    2563 server.go:87] "Starting device plugin registration server"
Sep 10 23:01:19.785649 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.762452    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 23:01:19.785649 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.762464    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 23:01:19.785649 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.762619    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 23:01:19.785649 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.762687    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 23:01:19.785649 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.762695    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 23:01:19.785649 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.763325    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 23:01:19.785649 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.763367    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-246.ec2.internal\" not found"
Sep 10 23:01:19.805950 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.805921    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 23:01:19.807240 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.807223    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 23:01:19.807325 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.807271    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 23:01:19.807325 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.807299    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 23:01:19.807429 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.807340    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 23:01:19.809639 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.809622    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:19.863028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.862994    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:19.864020 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.863999    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:19.864125 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.864031    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:19.864125 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.864042    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:19.864125 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.864065    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-246.ec2.internal"
Sep 10 23:01:19.872196 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.872125    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-246.ec2.internal"
Sep 10 23:01:19.872196 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.872152    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-246.ec2.internal\": node \"ip-10-0-130-246.ec2.internal\" not found"
Sep 10 23:01:19.897377 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.897353    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-246.ec2.internal\" not found"
Sep 10 23:01:19.908437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.908414    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-246.ec2.internal"]
Sep 10 23:01:19.908539 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.908478    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:19.909356 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.909343    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:19.909413 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.909367    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:19.909413 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.909377    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:19.911623 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.911611    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:19.911728 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.911713    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:19.911771 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.911740    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:19.911921 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.911905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e2d8607ec87ae4c7192d16bd654f102-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal\" (UID: \"3e2d8607ec87ae4c7192d16bd654f102\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:19.911986 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.911927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e2d8607ec87ae4c7192d16bd654f102-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal\" (UID: \"3e2d8607ec87ae4c7192d16bd654f102\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:19.912363 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.912348    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:19.912363 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.912356    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:19.912474 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.912374    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:19.912474 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.912384    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:19.912587 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.912375    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:19.912587 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.912534    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:19.914785 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.914768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:19.914867 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.914798    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 23:01:19.915424 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.915411    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 23:01:19.915487 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.915437    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 23:01:19.915487 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:19.915447    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeHasSufficientPID"
Sep 10 23:01:19.939178 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.939156    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-246.ec2.internal\" not found" node="ip-10-0-130-246.ec2.internal"
Sep 10 23:01:19.943591 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.943576    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-246.ec2.internal\" not found" node="ip-10-0-130-246.ec2.internal"
Sep 10 23:01:19.998180 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:19.998156    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-246.ec2.internal\" not found"
Sep 10 23:01:20.012451 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.012429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e2d8607ec87ae4c7192d16bd654f102-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal\" (UID: \"3e2d8607ec87ae4c7192d16bd654f102\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:20.012549 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.012464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e2d8607ec87ae4c7192d16bd654f102-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal\" (UID: \"3e2d8607ec87ae4c7192d16bd654f102\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:20.012549 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.012501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8749419d048b9a7948ba1036740398d2-config\") pod \"kube-apiserver-proxy-ip-10-0-130-246.ec2.internal\" (UID: \"8749419d048b9a7948ba1036740398d2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:20.012615 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.012590    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3e2d8607ec87ae4c7192d16bd654f102-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal\" (UID: \"3e2d8607ec87ae4c7192d16bd654f102\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:20.012658 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.012641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e2d8607ec87ae4c7192d16bd654f102-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal\" (UID: \"3e2d8607ec87ae4c7192d16bd654f102\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:20.098642 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.098604    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-246.ec2.internal\" not found"
Sep 10 23:01:20.112944 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.112920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8749419d048b9a7948ba1036740398d2-config\") pod \"kube-apiserver-proxy-ip-10-0-130-246.ec2.internal\" (UID: \"8749419d048b9a7948ba1036740398d2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:20.113011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.112973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8749419d048b9a7948ba1036740398d2-config\") pod \"kube-apiserver-proxy-ip-10-0-130-246.ec2.internal\" (UID: \"8749419d048b9a7948ba1036740398d2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:20.199423 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.199350    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-246.ec2.internal\" not found"
Sep 10 23:01:20.242883 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.242863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:20.246020 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.246004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:20.299681 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.299648    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-246.ec2.internal\" not found"
Sep 10 23:01:20.400207 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.400183    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-246.ec2.internal\" not found"
Sep 10 23:01:20.501124 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.501024    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-246.ec2.internal\" not found"
Sep 10 23:01:20.533773 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.533747    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:20.563407 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.563388    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:20.611080 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.611057    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:20.623826 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.623802    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 23:01:20.625176 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.625162    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-246.ec2.internal"
Sep 10 23:01:20.633701 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.633684    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 23:01:20.642223 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.642200    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 23:01:20.642365 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.642344    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 23:01:20.642421 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.642385    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 23:01:20.642421 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.642385    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 23:01:20.685140 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.685099    2563 apiserver.go:52] "Watching apiserver"
Sep 10 23:01:20.701682 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.701651    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 23:01:20.702555 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.702531    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-whf9r","openshift-dns/node-resolver-5j59t","openshift-image-registry/node-ca-n6rbk","openshift-multus/multus-ctqkx","openshift-network-diagnostics/network-check-target-kr684","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal","openshift-multus/multus-additional-cni-plugins-md4pm","openshift-multus/network-metrics-daemon-v8sx8","openshift-network-operator/iptables-alerter-vjp6c","openshift-ovn-kubernetes/ovnkube-node-b88ks","kube-system/global-pull-secret-syncer-xdms4","kube-system/konnectivity-agent-pgsk2","kube-system/kube-apiserver-proxy-ip-10-0-130-246.ec2.internal"]
Sep 10 23:01:20.705098 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.705080    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.707298 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.707278    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.707417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.707347    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 23:01:20.707417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.707276    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 23:01:20.707417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.707377    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 23:01:20.707584 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.707468    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 23:01:20.707643 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.707631    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k9sdg\""
Sep 10 23:01:20.709183 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.709168    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5ktjc\""
Sep 10 23:01:20.709419 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.709404    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 23:01:20.709492 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.709427    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 23:01:20.709571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.709539    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 23:01:20.709571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.709559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n6rbk"
Sep 10 23:01:20.711807 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.711752    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 23:01:20.711909 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.711842    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d9zbd\""
Sep 10 23:01:20.712086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.712067    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 23:01:20.712207 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.712191    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 23:01:20.712452 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.712433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:20.712634 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.712601    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:20.714779 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.714758    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.716368 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.716450 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-run-netns\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.716450 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mpth\" (UniqueName: \"kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth\") pod \"network-check-target-kr684\" (UID: \"34885b0c-c971-4056-a091-1764a294ea66\") " pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:20.716450 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-multus-cni-dir\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.716450 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-cnibin\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.716450 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21f78481-f131-45b0-ba1d-498e4f3d76bd-multus-daemon-config\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.716687 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-var-lib-cni-bin\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.716687 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716506    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-cnibin\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.716687 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bc7dd59a-fbd1-49b2-b886-ee2429896ddc-serviceca\") pod \"node-ca-n6rbk\" (UID: \"bc7dd59a-fbd1-49b2-b886-ee2429896ddc\") " pod="openshift-image-registry/node-ca-n6rbk"
Sep 10 23:01:20.716687 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-etc-kubernetes\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.716687 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716572    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-tuning-conf-dir\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.716687 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc7dd59a-fbd1-49b2-b886-ee2429896ddc-host\") pod \"node-ca-n6rbk\" (UID: \"bc7dd59a-fbd1-49b2-b886-ee2429896ddc\") " pod="openshift-image-registry/node-ca-n6rbk"
Sep 10 23:01:20.716687 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-var-lib-cni-multus\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.716687 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-run-k8s-cni-cncf-io\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.716687 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-system-cni-dir\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-run-multus-certs\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xs9x\" (UniqueName: \"kubernetes.io/projected/21f78481-f131-45b0-ba1d-498e4f3d76bd-kube-api-access-4xs9x\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716760    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-cni-binary-copy\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716737    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716785    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6hsz\" (UniqueName: \"kubernetes.io/projected/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-kube-api-access-t6hsz\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-var-lib-kubelet\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-hostroot\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-multus-socket-dir-parent\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716873    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-multus-conf-dir\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pjls\" (UniqueName: \"kubernetes.io/projected/bc7dd59a-fbd1-49b2-b886-ee2429896ddc-kube-api-access-2pjls\") pod \"node-ca-n6rbk\" (UID: \"bc7dd59a-fbd1-49b2-b886-ee2429896ddc\") " pod="openshift-image-registry/node-ca-n6rbk"
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21f78481-f131-45b0-ba1d-498e4f3d76bd-cni-binary-copy\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.716994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-system-cni-dir\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.717018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.717086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.717057    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f4g4m\""
Sep 10 23:01:20.717655 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.717172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.717655 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.717206    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 23:01:20.719177 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.719162    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vlw2p\""
Sep 10 23:01:20.719390 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.719373    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5j59t"
Sep 10 23:01:20.719456 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.719405    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:01:20.719456 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.719408    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 23:01:20.721767 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.721751    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 23:01:20.721839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.721769    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 23:01:20.721885 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.721871    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4ms9j\""
Sep 10 23:01:20.721979 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.721964    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 23:01:20.722930 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.722909    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:20.723029 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.722976    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:20.725178 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.725164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vjp6c"
Sep 10 23:01:20.727351 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.727331    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-78gpp\""
Sep 10 23:01:20.727428 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.727378    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 23:01:20.727489 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.727473    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 23:01:20.727571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.727557    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:01:20.727663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.727648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.730228 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.730211    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:20.730303 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.730240    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4g5xv\""
Sep 10 23:01:20.730303 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.730281    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:20.730303 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.730299    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 23:01:20.730470 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.730325    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 23:01:20.730470 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.730338    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 23:01:20.730470 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.730387    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 23:01:20.730470 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.730302    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 23:01:20.730657 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.730498    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 23:01:20.732472 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.732456    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pgsk2"
Sep 10 23:01:20.734517 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.734503    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 23:01:20.734585 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.734519    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v9pzx\""
Sep 10 23:01:20.734736 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.734724    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 23:01:20.740985 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.738398    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 22:56:19 +0000 UTC" deadline="2028-04-19 00:39:07.910089936 +0000 UTC"
Sep 10 23:01:20.740985 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.738432    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14065h37m47.171660584s"
Sep 10 23:01:20.742567 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.742552    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n7fbl"
Sep 10 23:01:20.750703 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.750681    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n7fbl"
Sep 10 23:01:20.811677 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.811657    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 23:01:20.817850 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.817833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-run-multus-certs\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.817901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.817856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xs9x\" (UniqueName: \"kubernetes.io/projected/21f78481-f131-45b0-ba1d-498e4f3d76bd-kube-api-access-4xs9x\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.817901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.817876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6hsz\" (UniqueName: \"kubernetes.io/projected/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-kube-api-access-t6hsz\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.817901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.817894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-etc-openvswitch\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.818025 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.817909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:20.818025 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.817924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-modprobe-d\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.818025 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.817939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-var-lib-kubelet\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.818025 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.817953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-cni-netd\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.818025 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.817955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-run-multus-certs\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.818025 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818004    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-var-lib-kubelet\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.818233 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/246cb358-713a-4429-aac2-8853a3d0dbdd-agent-certs\") pod \"konnectivity-agent-pgsk2\" (UID: \"246cb358-713a-4429-aac2-8853a3d0dbdd\") " pod="kube-system/konnectivity-agent-pgsk2"
Sep 10 23:01:20.818233 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-run-netns\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.818233 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld2mr\" (UniqueName: \"kubernetes.io/projected/df881283-5594-424d-98e4-e75fec3b7d19-kube-api-access-ld2mr\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.818233 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-tuned\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.818233 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02e28b7f-e193-411b-875c-e475adab6c34-hosts-file\") pod \"node-resolver-5j59t\" (UID: \"02e28b7f-e193-411b-875c-e475adab6c34\") " pod="openshift-dns/node-resolver-5j59t"
Sep 10 23:01:20.818233 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21f78481-f131-45b0-ba1d-498e4f3d76bd-cni-binary-copy\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.818233 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-system-cni-dir\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.818233 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.818233 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-run-ovn-kubernetes\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.818233 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab0193de-c9ef-4b5c-a865-bf880e0761d0-dbus\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:20.818603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-system-cni-dir\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.818603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-sysconfig\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.818603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07825aef-03b4-4691-8166-1cf8462cc3fc-tmp\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.818603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpth\" (UniqueName: \"kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth\") pod \"network-check-target-kr684\" (UID: \"34885b0c-c971-4056-a091-1764a294ea66\") " pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:20.818603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-socket-dir\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.818603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818447    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79gfh\" (UniqueName: \"kubernetes.io/projected/54f0cb94-e7d5-4b86-8602-dd8b46138dab-kube-api-access-79gfh\") pod \"network-metrics-daemon-v8sx8\" (UID: \"54f0cb94-e7d5-4b86-8602-dd8b46138dab\") " pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:20.818603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818514    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-cnibin\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.818603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-var-lib-openvswitch\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.818603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bc7dd59a-fbd1-49b2-b886-ee2429896ddc-serviceca\") pod \"node-ca-n6rbk\" (UID: \"bc7dd59a-fbd1-49b2-b886-ee2429896ddc\") " pod="openshift-image-registry/node-ca-n6rbk"
Sep 10 23:01:20.818603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-cnibin\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.818603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-etc-kubernetes\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.818923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818617    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/df881283-5594-424d-98e4-e75fec3b7d19-env-overrides\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.818923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-kubernetes\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.818923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818662    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-systemd\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.818923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-host\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.818923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-etc-kubernetes\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.818923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818729    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-node-log\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.818923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/df881283-5594-424d-98e4-e75fec3b7d19-ovn-node-metrics-cert\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.818923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-sysctl-conf\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.818923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02e28b7f-e193-411b-875c-e475adab6c34-tmp-dir\") pod \"node-resolver-5j59t\" (UID: \"02e28b7f-e193-411b-875c-e475adab6c34\") " pod="openshift-dns/node-resolver-5j59t"
Sep 10 23:01:20.818923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs\") pod \"network-metrics-daemon-v8sx8\" (UID: \"54f0cb94-e7d5-4b86-8602-dd8b46138dab\") " pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:20.818923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-run-k8s-cni-cncf-io\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.818923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-system-cni-dir\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.818923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-cni-binary-copy\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818928    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-run-k8s-cni-cncf-io\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-hostroot\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-system-cni-dir\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.818996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-run-systemd\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-hostroot\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-cni-bin\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab0193de-c9ef-4b5c-a865-bf880e0761d0-kubelet-config\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bc7dd59a-fbd1-49b2-b886-ee2429896ddc-serviceca\") pod \"node-ca-n6rbk\" (UID: \"bc7dd59a-fbd1-49b2-b886-ee2429896ddc\") " pod="openshift-image-registry/node-ca-n6rbk"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-multus-socket-dir-parent\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-multus-conf-dir\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-multus-socket-dir-parent\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-log-socket\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/df881283-5594-424d-98e4-e75fec3b7d19-ovnkube-script-lib\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-multus-conf-dir\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-device-dir\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.819345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-var-lib-kubelet\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pjls\" (UniqueName: \"kubernetes.io/projected/bc7dd59a-fbd1-49b2-b886-ee2429896ddc-kube-api-access-2pjls\") pod \"node-ca-n6rbk\" (UID: \"bc7dd59a-fbd1-49b2-b886-ee2429896ddc\") " pod="openshift-image-registry/node-ca-n6rbk"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-slash\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-registration-dir\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21f78481-f131-45b0-ba1d-498e4f3d76bd-cni-binary-copy\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819348    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-etc-selinux\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-sysctl-d\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-run-netns\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-multus-cni-dir\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-cnibin\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21f78481-f131-45b0-ba1d-498e4f3d76bd-multus-daemon-config\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-systemd-units\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-cnibin\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-lib-modules\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-run-netns\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc4m5\" (UniqueName: \"kubernetes.io/projected/02e28b7f-e193-411b-875c-e475adab6c34-kube-api-access-pc4m5\") pod \"node-resolver-5j59t\" (UID: \"02e28b7f-e193-411b-875c-e475adab6c34\") " pod="openshift-dns/node-resolver-5j59t"
Sep 10 23:01:20.820081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-var-lib-cni-bin\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-multus-cni-dir\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/df881283-5594-424d-98e4-e75fec3b7d19-ovnkube-config\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/246cb358-713a-4429-aac2-8853a3d0dbdd-konnectivity-ca\") pod \"konnectivity-agent-pgsk2\" (UID: \"246cb358-713a-4429-aac2-8853a3d0dbdd\") " pod="kube-system/konnectivity-agent-pgsk2"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-var-lib-cni-bin\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-sys-fs\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-run\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-sys\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7-host-slash\") pod \"iptables-alerter-vjp6c\" (UID: \"f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7\") " pod="openshift-network-operator/iptables-alerter-vjp6c"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-tuning-conf-dir\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-kubelet\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-run-openvswitch\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-run-ovn\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21f78481-f131-45b0-ba1d-498e4f3d76bd-multus-daemon-config\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsgtd\" (UniqueName: \"kubernetes.io/projected/2863dbdf-da97-48ce-8950-d0eeabdd9039-kube-api-access-hsgtd\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-tuning-conf-dir\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.820631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddvpw\" (UniqueName: \"kubernetes.io/projected/f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7-kube-api-access-ddvpw\") pod \"iptables-alerter-vjp6c\" (UID: \"f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7\") " pod="openshift-network-operator/iptables-alerter-vjp6c"
Sep 10 23:01:20.821103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc7dd59a-fbd1-49b2-b886-ee2429896ddc-host\") pod \"node-ca-n6rbk\" (UID: \"bc7dd59a-fbd1-49b2-b886-ee2429896ddc\") " pod="openshift-image-registry/node-ca-n6rbk"
Sep 10 23:01:20.821103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-var-lib-cni-multus\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.821103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.819995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf526\" (UniqueName: \"kubernetes.io/projected/07825aef-03b4-4691-8166-1cf8462cc3fc-kube-api-access-hf526\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.821103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.820020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7-iptables-alerter-script\") pod \"iptables-alerter-vjp6c\" (UID: \"f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7\") " pod="openshift-network-operator/iptables-alerter-vjp6c"
Sep 10 23:01:20.821103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.820026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc7dd59a-fbd1-49b2-b886-ee2429896ddc-host\") pod \"node-ca-n6rbk\" (UID: \"bc7dd59a-fbd1-49b2-b886-ee2429896ddc\") " pod="openshift-image-registry/node-ca-n6rbk"
Sep 10 23:01:20.821103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.820033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21f78481-f131-45b0-ba1d-498e4f3d76bd-host-var-lib-cni-multus\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.821103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.820041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-cni-binary-copy\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.821103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.820053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.821103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.820074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.827679 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.827652    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 23:01:20.828085 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.828067    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:20.828154 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.828090    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:20.828154 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.828103    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5mpth for pod openshift-network-diagnostics/network-check-target-kr684: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:20.828238 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.828204    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth podName:34885b0c-c971-4056-a091-1764a294ea66 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:21.328185602 +0000 UTC m=+2.019762785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5mpth" (UniqueName: "kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth") pod "network-check-target-kr684" (UID: "34885b0c-c971-4056-a091-1764a294ea66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:20.831130 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.831100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pjls\" (UniqueName: \"kubernetes.io/projected/bc7dd59a-fbd1-49b2-b886-ee2429896ddc-kube-api-access-2pjls\") pod \"node-ca-n6rbk\" (UID: \"bc7dd59a-fbd1-49b2-b886-ee2429896ddc\") " pod="openshift-image-registry/node-ca-n6rbk"
Sep 10 23:01:20.831229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.831161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xs9x\" (UniqueName: \"kubernetes.io/projected/21f78481-f131-45b0-ba1d-498e4f3d76bd-kube-api-access-4xs9x\") pod \"multus-ctqkx\" (UID: \"21f78481-f131-45b0-ba1d-498e4f3d76bd\") " pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:20.831229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.831187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6hsz\" (UniqueName: \"kubernetes.io/projected/0b14fb8d-3467-4ca7-9200-d6ae4692aa22-kube-api-access-t6hsz\") pod \"multus-additional-cni-plugins-md4pm\" (UID: \"0b14fb8d-3467-4ca7-9200-d6ae4692aa22\") " pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:20.920570 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gfh\" (UniqueName: \"kubernetes.io/projected/54f0cb94-e7d5-4b86-8602-dd8b46138dab-kube-api-access-79gfh\") pod \"network-metrics-daemon-v8sx8\" (UID: \"54f0cb94-e7d5-4b86-8602-dd8b46138dab\") " pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:20.920746 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-var-lib-openvswitch\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.920746 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/df881283-5594-424d-98e4-e75fec3b7d19-env-overrides\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.920746 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-kubernetes\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.920746 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-var-lib-openvswitch\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.920746 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920679    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-kubernetes\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.920746 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-systemd\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.920746 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-host\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-node-log\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920789    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-systemd\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-host\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-node-log\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/df881283-5594-424d-98e4-e75fec3b7d19-ovn-node-metrics-cert\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-sysctl-conf\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02e28b7f-e193-411b-875c-e475adab6c34-tmp-dir\") pod \"node-resolver-5j59t\" (UID: \"02e28b7f-e193-411b-875c-e475adab6c34\") " pod="openshift-dns/node-resolver-5j59t"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs\") pod \"network-metrics-daemon-v8sx8\" (UID: \"54f0cb94-e7d5-4b86-8602-dd8b46138dab\") " pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.920971    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.921033    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs podName:54f0cb94-e7d5-4b86-8602-dd8b46138dab nodeName:}" failed. No retries permitted until 2026-09-10 23:01:21.421015001 +0000 UTC m=+2.112592179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs") pod "network-metrics-daemon-v8sx8" (UID: "54f0cb94-e7d5-4b86-8602-dd8b46138dab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-sysctl-conf\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.920964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-run-systemd\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921134    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-cni-bin\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921143    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/df881283-5594-424d-98e4-e75fec3b7d19-env-overrides\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-run-systemd\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab0193de-c9ef-4b5c-a865-bf880e0761d0-kubelet-config\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-cni-bin\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-log-socket\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921214    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ab0193de-c9ef-4b5c-a865-bf880e0761d0-kubelet-config\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/df881283-5594-424d-98e4-e75fec3b7d19-ovnkube-script-lib\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02e28b7f-e193-411b-875c-e475adab6c34-tmp-dir\") pod \"node-resolver-5j59t\" (UID: \"02e28b7f-e193-411b-875c-e475adab6c34\") " pod="openshift-dns/node-resolver-5j59t"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-log-socket\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-device-dir\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-var-lib-kubelet\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-device-dir\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-var-lib-kubelet\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-slash\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-registration-dir\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-etc-selinux\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-registration-dir\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-slash\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.921966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-sysctl-d\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921557    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-etc-selinux\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-systemd-units\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-lib-modules\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc4m5\" (UniqueName: \"kubernetes.io/projected/02e28b7f-e193-411b-875c-e475adab6c34-kube-api-access-pc4m5\") pod \"node-resolver-5j59t\" (UID: \"02e28b7f-e193-411b-875c-e475adab6c34\") " pod="openshift-dns/node-resolver-5j59t"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-systemd-units\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921668    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-sysctl-d\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/df881283-5594-424d-98e4-e75fec3b7d19-ovnkube-config\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/246cb358-713a-4429-aac2-8853a3d0dbdd-konnectivity-ca\") pod \"konnectivity-agent-pgsk2\" (UID: \"246cb358-713a-4429-aac2-8853a3d0dbdd\") " pod="kube-system/konnectivity-agent-pgsk2"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-lib-modules\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/df881283-5594-424d-98e4-e75fec3b7d19-ovnkube-script-lib\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-sys-fs\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-run\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-run\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-sys\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7-host-slash\") pod \"iptables-alerter-vjp6c\" (UID: \"f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7\") " pod="openshift-network-operator/iptables-alerter-vjp6c"
Sep 10 23:01:20.922577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-sys\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-sys-fs\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-kubelet\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-run-openvswitch\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7-host-slash\") pod \"iptables-alerter-vjp6c\" (UID: \"f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7\") " pod="openshift-network-operator/iptables-alerter-vjp6c"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.921995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-kubelet\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-run-ovn\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-run-openvswitch\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-run-ovn\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/df881283-5594-424d-98e4-e75fec3b7d19-ovnkube-config\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/246cb358-713a-4429-aac2-8853a3d0dbdd-konnectivity-ca\") pod \"konnectivity-agent-pgsk2\" (UID: \"246cb358-713a-4429-aac2-8853a3d0dbdd\") " pod="kube-system/konnectivity-agent-pgsk2"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsgtd\" (UniqueName: \"kubernetes.io/projected/2863dbdf-da97-48ce-8950-d0eeabdd9039-kube-api-access-hsgtd\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddvpw\" (UniqueName: \"kubernetes.io/projected/f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7-kube-api-access-ddvpw\") pod \"iptables-alerter-vjp6c\" (UID: \"f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7\") " pod="openshift-network-operator/iptables-alerter-vjp6c"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf526\" (UniqueName: \"kubernetes.io/projected/07825aef-03b4-4691-8166-1cf8462cc3fc-kube-api-access-hf526\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7-iptables-alerter-script\") pod \"iptables-alerter-vjp6c\" (UID: \"f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7\") " pod="openshift-network-operator/iptables-alerter-vjp6c"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-etc-openvswitch\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-modprobe-d\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.923437 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.922845    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-cni-netd\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:20.922908    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret podName:ab0193de-c9ef-4b5c-a865-bf880e0761d0 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:21.42288947 +0000 UTC m=+2.114466648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret") pod "global-pull-secret-syncer-xdms4" (UID: "ab0193de-c9ef-4b5c-a865-bf880e0761d0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-cni-netd\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922924    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-modprobe-d\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/246cb358-713a-4429-aac2-8853a3d0dbdd-agent-certs\") pod \"konnectivity-agent-pgsk2\" (UID: \"246cb358-713a-4429-aac2-8853a3d0dbdd\") " pod="kube-system/konnectivity-agent-pgsk2"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-run-netns\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld2mr\" (UniqueName: \"kubernetes.io/projected/df881283-5594-424d-98e4-e75fec3b7d19-kube-api-access-ld2mr\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.922980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-etc-openvswitch\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-tuned\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02e28b7f-e193-411b-875c-e475adab6c34-hosts-file\") pod \"node-resolver-5j59t\" (UID: \"02e28b7f-e193-411b-875c-e475adab6c34\") " pod="openshift-dns/node-resolver-5j59t"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-run-netns\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-run-ovn-kubernetes\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923135    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df881283-5594-424d-98e4-e75fec3b7d19-host-run-ovn-kubernetes\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab0193de-c9ef-4b5c-a865-bf880e0761d0-dbus\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-sysconfig\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ab0193de-c9ef-4b5c-a865-bf880e0761d0-dbus\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:20.924126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07825aef-03b4-4691-8166-1cf8462cc3fc-tmp\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.924694 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923252    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-socket-dir\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.924694 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02e28b7f-e193-411b-875c-e475adab6c34-hosts-file\") pod \"node-resolver-5j59t\" (UID: \"02e28b7f-e193-411b-875c-e475adab6c34\") " pod="openshift-dns/node-resolver-5j59t"
Sep 10 23:01:20.924694 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7-iptables-alerter-script\") pod \"iptables-alerter-vjp6c\" (UID: \"f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7\") " pod="openshift-network-operator/iptables-alerter-vjp6c"
Sep 10 23:01:20.924694 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2863dbdf-da97-48ce-8950-d0eeabdd9039-socket-dir\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.924694 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.923388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-sysconfig\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.924694 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.924102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/df881283-5594-424d-98e4-e75fec3b7d19-ovn-node-metrics-cert\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.925144 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.925122    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/07825aef-03b4-4691-8166-1cf8462cc3fc-etc-tuned\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.925672 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.925655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07825aef-03b4-4691-8166-1cf8462cc3fc-tmp\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.925867 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.925849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/246cb358-713a-4429-aac2-8853a3d0dbdd-agent-certs\") pod \"konnectivity-agent-pgsk2\" (UID: \"246cb358-713a-4429-aac2-8853a3d0dbdd\") " pod="kube-system/konnectivity-agent-pgsk2"
Sep 10 23:01:20.929773 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.929753    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gfh\" (UniqueName: \"kubernetes.io/projected/54f0cb94-e7d5-4b86-8602-dd8b46138dab-kube-api-access-79gfh\") pod \"network-metrics-daemon-v8sx8\" (UID: \"54f0cb94-e7d5-4b86-8602-dd8b46138dab\") " pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:20.930957 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.930932    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc4m5\" (UniqueName: \"kubernetes.io/projected/02e28b7f-e193-411b-875c-e475adab6c34-kube-api-access-pc4m5\") pod \"node-resolver-5j59t\" (UID: \"02e28b7f-e193-411b-875c-e475adab6c34\") " pod="openshift-dns/node-resolver-5j59t"
Sep 10 23:01:20.931345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.931327    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf526\" (UniqueName: \"kubernetes.io/projected/07825aef-03b4-4691-8166-1cf8462cc3fc-kube-api-access-hf526\") pod \"tuned-whf9r\" (UID: \"07825aef-03b4-4691-8166-1cf8462cc3fc\") " pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:20.931468 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.931449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsgtd\" (UniqueName: \"kubernetes.io/projected/2863dbdf-da97-48ce-8950-d0eeabdd9039-kube-api-access-hsgtd\") pod \"aws-ebs-csi-driver-node-7kg25\" (UID: \"2863dbdf-da97-48ce-8950-d0eeabdd9039\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:20.931613 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.931598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddvpw\" (UniqueName: \"kubernetes.io/projected/f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7-kube-api-access-ddvpw\") pod \"iptables-alerter-vjp6c\" (UID: \"f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7\") " pod="openshift-network-operator/iptables-alerter-vjp6c"
Sep 10 23:01:20.931827 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.931811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld2mr\" (UniqueName: \"kubernetes.io/projected/df881283-5594-424d-98e4-e75fec3b7d19-kube-api-access-ld2mr\") pod \"ovnkube-node-b88ks\" (UID: \"df881283-5594-424d-98e4-e75fec3b7d19\") " pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:20.938994 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:20.938974    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:21.042762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.042692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ctqkx"
Sep 10 23:01:21.058244 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:21.058211    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8749419d048b9a7948ba1036740398d2.slice/crio-a22d08e6108012b2b180efe5047b1b0ccf24596d1029740715eee31ce5d0851d WatchSource:0}: Error finding container a22d08e6108012b2b180efe5047b1b0ccf24596d1029740715eee31ce5d0851d: Status 404 returned error can't find the container with id a22d08e6108012b2b180efe5047b1b0ccf24596d1029740715eee31ce5d0851d
Sep 10 23:01:21.058469 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:21.058457    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f78481_f131_45b0_ba1d_498e4f3d76bd.slice/crio-fc6e60e0e30177f0205a66c9398bb269baf0a268c5321a8b7d8d51782535553a WatchSource:0}: Error finding container fc6e60e0e30177f0205a66c9398bb269baf0a268c5321a8b7d8d51782535553a: Status 404 returned error can't find the container with id fc6e60e0e30177f0205a66c9398bb269baf0a268c5321a8b7d8d51782535553a
Sep 10 23:01:21.060698 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.060674    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-md4pm"
Sep 10 23:01:21.065283 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.064918    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:01:21.073109 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:21.073085    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b14fb8d_3467_4ca7_9200_d6ae4692aa22.slice/crio-941e2ca4f1cd480e57628066c8daddb5457a058c7ef1ee5c7913085abbeb6b50 WatchSource:0}: Error finding container 941e2ca4f1cd480e57628066c8daddb5457a058c7ef1ee5c7913085abbeb6b50: Status 404 returned error can't find the container with id 941e2ca4f1cd480e57628066c8daddb5457a058c7ef1ee5c7913085abbeb6b50
Sep 10 23:01:21.074449 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.074431    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n6rbk"
Sep 10 23:01:21.082937 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:21.082911    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7dd59a_fbd1_49b2_b886_ee2429896ddc.slice/crio-7fe703c7244282cbf67d0039c3d9c0d04d4d3723147bbd7df03741790dcef134 WatchSource:0}: Error finding container 7fe703c7244282cbf67d0039c3d9c0d04d4d3723147bbd7df03741790dcef134: Status 404 returned error can't find the container with id 7fe703c7244282cbf67d0039c3d9c0d04d4d3723147bbd7df03741790dcef134
Sep 10 23:01:21.087633 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.087616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25"
Sep 10 23:01:21.093558 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.093534    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-whf9r"
Sep 10 23:01:21.095562 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:21.095540    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2863dbdf_da97_48ce_8950_d0eeabdd9039.slice/crio-2c9148e4edc3a382314586c162d1f466452560d3ab0747c03cff99e4be760731 WatchSource:0}: Error finding container 2c9148e4edc3a382314586c162d1f466452560d3ab0747c03cff99e4be760731: Status 404 returned error can't find the container with id 2c9148e4edc3a382314586c162d1f466452560d3ab0747c03cff99e4be760731
Sep 10 23:01:21.101825 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:21.101672    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07825aef_03b4_4691_8166_1cf8462cc3fc.slice/crio-c0fbfdd00a65820e10210c8500f1b205af06f873bf60d473c6e8452b5bd5a8a1 WatchSource:0}: Error finding container c0fbfdd00a65820e10210c8500f1b205af06f873bf60d473c6e8452b5bd5a8a1: Status 404 returned error can't find the container with id c0fbfdd00a65820e10210c8500f1b205af06f873bf60d473c6e8452b5bd5a8a1
Sep 10 23:01:21.127580 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:21.127546    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2d8607ec87ae4c7192d16bd654f102.slice/crio-c85c4539a64f4e537c03ae94b259593c8868f8da39364d0cca07d5ed3279c963 WatchSource:0}: Error finding container c85c4539a64f4e537c03ae94b259593c8868f8da39364d0cca07d5ed3279c963: Status 404 returned error can't find the container with id c85c4539a64f4e537c03ae94b259593c8868f8da39364d0cca07d5ed3279c963
Sep 10 23:01:21.128698 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.128679    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5j59t"
Sep 10 23:01:21.136492 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:21.136473    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e28b7f_e193_411b_875c_e475adab6c34.slice/crio-1ef0e59e05131f49d05c7875dbd570f71fc774737af5578d185e07ea3f4ddd26 WatchSource:0}: Error finding container 1ef0e59e05131f49d05c7875dbd570f71fc774737af5578d185e07ea3f4ddd26: Status 404 returned error can't find the container with id 1ef0e59e05131f49d05c7875dbd570f71fc774737af5578d185e07ea3f4ddd26
Sep 10 23:01:21.140765 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.140747    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vjp6c"
Sep 10 23:01:21.150074 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:21.150044    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b7c54d_0f06_47e3_b7c8_0d6464f7f9b7.slice/crio-47a035b5ea4753155a6d09ae2e8ead065bc8a0db439a2d7dc430b15fe5a3aab2 WatchSource:0}: Error finding container 47a035b5ea4753155a6d09ae2e8ead065bc8a0db439a2d7dc430b15fe5a3aab2: Status 404 returned error can't find the container with id 47a035b5ea4753155a6d09ae2e8ead065bc8a0db439a2d7dc430b15fe5a3aab2
Sep 10 23:01:21.157253 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.157232    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:21.162004 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.161982    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pgsk2"
Sep 10 23:01:21.166576 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:21.166554    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf881283_5594_424d_98e4_e75fec3b7d19.slice/crio-93abf04c55d34aa34de371c6cdea7d8e068ec43cf4773e921a6afd40eb8a2137 WatchSource:0}: Error finding container 93abf04c55d34aa34de371c6cdea7d8e068ec43cf4773e921a6afd40eb8a2137: Status 404 returned error can't find the container with id 93abf04c55d34aa34de371c6cdea7d8e068ec43cf4773e921a6afd40eb8a2137
Sep 10 23:01:21.171339 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:21.171318    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246cb358_713a_4429_aac2_8853a3d0dbdd.slice/crio-7f4424287f5a8456ffe455d596e79d915aa0bb68016e94bf15b8474830a1211b WatchSource:0}: Error finding container 7f4424287f5a8456ffe455d596e79d915aa0bb68016e94bf15b8474830a1211b: Status 404 returned error can't find the container with id 7f4424287f5a8456ffe455d596e79d915aa0bb68016e94bf15b8474830a1211b
Sep 10 23:01:21.172917 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:21.172896    2563 kuberuntime_manager.go:1459] "Unhandled Error" err="container konnectivity-agent start failed in pod konnectivity-agent-pgsk2_kube-system(246cb358-713a-4429-aac2-8853a3d0dbdd): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 10 23:01:21.172988 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:21.172933    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ErrImagePull: \"pull QPS exceeded\"" pod="kube-system/konnectivity-agent-pgsk2" podUID="246cb358-713a-4429-aac2-8853a3d0dbdd"
Sep 10 23:01:21.427571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.427532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpth\" (UniqueName: \"kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth\") pod \"network-check-target-kr684\" (UID: \"34885b0c-c971-4056-a091-1764a294ea66\") " pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:21.427571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.427579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs\") pod \"network-metrics-daemon-v8sx8\" (UID: \"54f0cb94-e7d5-4b86-8602-dd8b46138dab\") " pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:21.427830 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.427622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:21.427830 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:21.427712    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:21.427830 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:21.427736    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:21.427830 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:21.427740    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:21.427830 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:21.427789    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs podName:54f0cb94-e7d5-4b86-8602-dd8b46138dab nodeName:}" failed. No retries permitted until 2026-09-10 23:01:22.42777577 +0000 UTC m=+3.119352932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs") pod "network-metrics-daemon-v8sx8" (UID: "54f0cb94-e7d5-4b86-8602-dd8b46138dab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:21.427830 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:21.427715    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:21.428139 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:21.427859    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret podName:ab0193de-c9ef-4b5c-a865-bf880e0761d0 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:22.42784543 +0000 UTC m=+3.119422593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret") pod "global-pull-secret-syncer-xdms4" (UID: "ab0193de-c9ef-4b5c-a865-bf880e0761d0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:21.428139 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:21.427746    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5mpth for pod openshift-network-diagnostics/network-check-target-kr684: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:21.428139 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:21.427899    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth podName:34885b0c-c971-4056-a091-1764a294ea66 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:22.427891897 +0000 UTC m=+3.119469060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mpth" (UniqueName: "kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth") pod "network-check-target-kr684" (UID: "34885b0c-c971-4056-a091-1764a294ea66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:21.753568 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.753430    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 22:56:20 +0000 UTC" deadline="2028-04-17 03:28:49.40535649 +0000 UTC"
Sep 10 23:01:21.753568 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.753515    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14020h27m27.651847571s"
Sep 10 23:01:21.829832 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.829774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks" event={"ID":"df881283-5594-424d-98e4-e75fec3b7d19","Type":"ContainerStarted","Data":"93abf04c55d34aa34de371c6cdea7d8e068ec43cf4773e921a6afd40eb8a2137"}
Sep 10 23:01:21.841806 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.841727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5j59t" event={"ID":"02e28b7f-e193-411b-875c-e475adab6c34","Type":"ContainerStarted","Data":"1ef0e59e05131f49d05c7875dbd570f71fc774737af5578d185e07ea3f4ddd26"}
Sep 10 23:01:21.855334 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.855254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal" event={"ID":"3e2d8607ec87ae4c7192d16bd654f102","Type":"ContainerStarted","Data":"c85c4539a64f4e537c03ae94b259593c8868f8da39364d0cca07d5ed3279c963"}
Sep 10 23:01:21.863338 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.863305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-whf9r" event={"ID":"07825aef-03b4-4691-8166-1cf8462cc3fc","Type":"ContainerStarted","Data":"c0fbfdd00a65820e10210c8500f1b205af06f873bf60d473c6e8452b5bd5a8a1"}
Sep 10 23:01:21.873477 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.873403    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25" event={"ID":"2863dbdf-da97-48ce-8950-d0eeabdd9039","Type":"ContainerStarted","Data":"2c9148e4edc3a382314586c162d1f466452560d3ab0747c03cff99e4be760731"}
Sep 10 23:01:21.883532 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.883399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n6rbk" event={"ID":"bc7dd59a-fbd1-49b2-b886-ee2429896ddc","Type":"ContainerStarted","Data":"7fe703c7244282cbf67d0039c3d9c0d04d4d3723147bbd7df03741790dcef134"}
Sep 10 23:01:21.895901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.895864    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-246.ec2.internal" event={"ID":"8749419d048b9a7948ba1036740398d2","Type":"ContainerStarted","Data":"a22d08e6108012b2b180efe5047b1b0ccf24596d1029740715eee31ce5d0851d"}
Sep 10 23:01:21.905025 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.904989    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pgsk2" event={"ID":"246cb358-713a-4429-aac2-8853a3d0dbdd","Type":"ContainerStarted","Data":"7f4424287f5a8456ffe455d596e79d915aa0bb68016e94bf15b8474830a1211b"}
Sep 10 23:01:21.908368 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:21.908335    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cb18802deb495c8d9b0ab2c3d8c3043b73eafbad562b04b71a3717870ccdff74\\\": ErrImagePull: pull QPS exceeded\"" pod="kube-system/konnectivity-agent-pgsk2" podUID="246cb358-713a-4429-aac2-8853a3d0dbdd"
Sep 10 23:01:21.920082 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.919870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vjp6c" event={"ID":"f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7","Type":"ContainerStarted","Data":"47a035b5ea4753155a6d09ae2e8ead065bc8a0db439a2d7dc430b15fe5a3aab2"}
Sep 10 23:01:21.927128 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.927102    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:21.941480 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.941319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md4pm" event={"ID":"0b14fb8d-3467-4ca7-9200-d6ae4692aa22","Type":"ContainerStarted","Data":"941e2ca4f1cd480e57628066c8daddb5457a058c7ef1ee5c7913085abbeb6b50"}
Sep 10 23:01:21.972779 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:21.972742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ctqkx" event={"ID":"21f78481-f131-45b0-ba1d-498e4f3d76bd","Type":"ContainerStarted","Data":"fc6e60e0e30177f0205a66c9398bb269baf0a268c5321a8b7d8d51782535553a"}
Sep 10 23:01:22.422559 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:22.422341    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:22.435542 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:22.435469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:22.435542 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:22.435535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpth\" (UniqueName: \"kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth\") pod \"network-check-target-kr684\" (UID: \"34885b0c-c971-4056-a091-1764a294ea66\") " pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:22.435778 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:22.435579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs\") pod \"network-metrics-daemon-v8sx8\" (UID: \"54f0cb94-e7d5-4b86-8602-dd8b46138dab\") " pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:22.435778 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:22.435718    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:22.435778 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:22.435780    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs podName:54f0cb94-e7d5-4b86-8602-dd8b46138dab nodeName:}" failed. No retries permitted until 2026-09-10 23:01:24.435759975 +0000 UTC m=+5.127337154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs") pod "network-metrics-daemon-v8sx8" (UID: "54f0cb94-e7d5-4b86-8602-dd8b46138dab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:22.436146 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:22.436126    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:22.436221 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:22.436179    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret podName:ab0193de-c9ef-4b5c-a865-bf880e0761d0 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:24.436165879 +0000 UTC m=+5.127743047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret") pod "global-pull-secret-syncer-xdms4" (UID: "ab0193de-c9ef-4b5c-a865-bf880e0761d0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:22.436284 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:22.436256    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:22.436284 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:22.436273    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:22.436284 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:22.436284    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5mpth for pod openshift-network-diagnostics/network-check-target-kr684: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:22.436427 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:22.436317    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth podName:34885b0c-c971-4056-a091-1764a294ea66 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:24.436306929 +0000 UTC m=+5.127884093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mpth" (UniqueName: "kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth") pod "network-check-target-kr684" (UID: "34885b0c-c971-4056-a091-1764a294ea66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:22.753840 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:22.753687    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 22:56:20 +0000 UTC" deadline="2028-03-23 23:24:56.780394267 +0000 UTC"
Sep 10 23:01:22.753840 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:22.753724    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13440h23m34.026674644s"
Sep 10 23:01:22.808990 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:22.808160    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:22.808990 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:22.808325    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:22.808990 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:22.808698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:22.808990 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:22.808786    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:22.808990 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:22.808865    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:22.808990 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:22.808948    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:23.670518 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:23.670425    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 23:01:24.450814 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:24.450775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:24.451269 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:24.450834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpth\" (UniqueName: \"kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth\") pod \"network-check-target-kr684\" (UID: \"34885b0c-c971-4056-a091-1764a294ea66\") " pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:24.451269 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:24.450865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs\") pod \"network-metrics-daemon-v8sx8\" (UID: \"54f0cb94-e7d5-4b86-8602-dd8b46138dab\") " pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:24.451269 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:24.450989    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:24.451269 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:24.451051    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs podName:54f0cb94-e7d5-4b86-8602-dd8b46138dab nodeName:}" failed. No retries permitted until 2026-09-10 23:01:28.451032795 +0000 UTC m=+9.142609970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs") pod "network-metrics-daemon-v8sx8" (UID: "54f0cb94-e7d5-4b86-8602-dd8b46138dab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:24.451535 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:24.451445    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:24.451535 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:24.451515    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret podName:ab0193de-c9ef-4b5c-a865-bf880e0761d0 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:28.451479994 +0000 UTC m=+9.143057158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret") pod "global-pull-secret-syncer-xdms4" (UID: "ab0193de-c9ef-4b5c-a865-bf880e0761d0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:24.451654 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:24.451604    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:24.451654 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:24.451623    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:24.451654 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:24.451636    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5mpth for pod openshift-network-diagnostics/network-check-target-kr684: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:24.451806 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:24.451674    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth podName:34885b0c-c971-4056-a091-1764a294ea66 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:28.451661851 +0000 UTC m=+9.143239028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mpth" (UniqueName: "kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth") pod "network-check-target-kr684" (UID: "34885b0c-c971-4056-a091-1764a294ea66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:24.808358 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:24.808212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:24.808358 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:24.808260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:24.808358 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:24.808350    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:24.808810 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:24.808695    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:24.808869 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:24.808804    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:24.808949 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:24.808885    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:26.807722 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:26.807681    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:26.808164 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:26.807786    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:26.808164 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:26.807797    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:26.808164 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:26.807820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:26.808164 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:26.807915    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:26.808164 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:26.808003    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:28.485313 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:28.485269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:28.486037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:28.485345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpth\" (UniqueName: \"kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth\") pod \"network-check-target-kr684\" (UID: \"34885b0c-c971-4056-a091-1764a294ea66\") " pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:28.486037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:28.485383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs\") pod \"network-metrics-daemon-v8sx8\" (UID: \"54f0cb94-e7d5-4b86-8602-dd8b46138dab\") " pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:28.486037 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:28.485416    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:28.486037 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:28.485481    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret podName:ab0193de-c9ef-4b5c-a865-bf880e0761d0 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:36.485461819 +0000 UTC m=+17.177038987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret") pod "global-pull-secret-syncer-xdms4" (UID: "ab0193de-c9ef-4b5c-a865-bf880e0761d0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:28.486037 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:28.485552    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:28.486037 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:28.485579    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:28.486037 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:28.485560    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:28.486037 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:28.485592    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5mpth for pod openshift-network-diagnostics/network-check-target-kr684: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:28.486037 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:28.485635    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs podName:54f0cb94-e7d5-4b86-8602-dd8b46138dab nodeName:}" failed. No retries permitted until 2026-09-10 23:01:36.485612182 +0000 UTC m=+17.177189359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs") pod "network-metrics-daemon-v8sx8" (UID: "54f0cb94-e7d5-4b86-8602-dd8b46138dab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:28.486037 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:28.485654    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth podName:34885b0c-c971-4056-a091-1764a294ea66 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:36.485643334 +0000 UTC m=+17.177220500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mpth" (UniqueName: "kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth") pod "network-check-target-kr684" (UID: "34885b0c-c971-4056-a091-1764a294ea66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:28.808135 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:28.808100    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:28.808304 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:28.808266    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:28.808304 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:28.808280    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:28.808629 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:28.808401    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:28.808629 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:28.808476    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:28.808629 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:28.808574    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:30.807858 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:30.807817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:30.808313 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:30.807941    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:30.808313 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:30.807971    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:30.808313 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:30.808002    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:30.808313 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:30.808104    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:30.808313 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:30.808191    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:32.807879 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:32.807846    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:32.808303 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:32.807972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:32.808303 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:32.807980    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:32.808303 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:32.808010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:32.808303 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:32.808124    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:32.808303 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:32.808197    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:34.808289 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:34.808244    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:34.808289 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:34.808278    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:34.808289 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:34.808290    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:34.808937 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:34.808368    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:34.808937 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:34.808435    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:34.808937 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:34.808536    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:36.541333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:36.541239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs\") pod \"network-metrics-daemon-v8sx8\" (UID: \"54f0cb94-e7d5-4b86-8602-dd8b46138dab\") " pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:36.541333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:36.541325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:36.541951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:36.541366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpth\" (UniqueName: \"kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth\") pod \"network-check-target-kr684\" (UID: \"34885b0c-c971-4056-a091-1764a294ea66\") " pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:36.541951 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:36.541456    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:36.541951 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:36.541514    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:36.541951 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:36.541538    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:36.541951 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:36.541556    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:36.541951 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:36.541569    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5mpth for pod openshift-network-diagnostics/network-check-target-kr684: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:36.541951 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:36.541540    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs podName:54f0cb94-e7d5-4b86-8602-dd8b46138dab nodeName:}" failed. No retries permitted until 2026-09-10 23:01:52.541523826 +0000 UTC m=+33.233100993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs") pod "network-metrics-daemon-v8sx8" (UID: "54f0cb94-e7d5-4b86-8602-dd8b46138dab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:36.541951 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:36.541629    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret podName:ab0193de-c9ef-4b5c-a865-bf880e0761d0 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:52.541613846 +0000 UTC m=+33.233191016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret") pod "global-pull-secret-syncer-xdms4" (UID: "ab0193de-c9ef-4b5c-a865-bf880e0761d0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:36.541951 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:36.541644    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth podName:34885b0c-c971-4056-a091-1764a294ea66 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:52.541634709 +0000 UTC m=+33.233211874 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mpth" (UniqueName: "kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth") pod "network-check-target-kr684" (UID: "34885b0c-c971-4056-a091-1764a294ea66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:36.808252 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:36.808215    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:36.808527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:36.808215    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:36.808527 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:36.808347    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:36.808527 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:36.808517    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:36.808712 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:36.808565    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:36.808712 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:36.808641    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:38.808152 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:38.808119    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:38.808613 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:38.808119    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:38.808613 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:38.808252    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:38.808613 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:38.808337    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:38.808613 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:38.808118    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:38.808613 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:38.808442    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:40.808137 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:40.807778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:40.808856 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:40.807831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:40.808856 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:40.808235    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:40.808856 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:40.807846    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:40.808856 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:40.808314    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:40.808856 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:40.808414    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:41.033539 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.033443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5j59t" event={"ID":"02e28b7f-e193-411b-875c-e475adab6c34","Type":"ContainerStarted","Data":"a2dc5d3c51ae432628a76a821551f9ab9d52fd5efa20490ffaccbb6c392f4f97"}
Sep 10 23:01:41.035021 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.034994    2563 generic.go:356] "Generic (PLEG): container finished" podID="3e2d8607ec87ae4c7192d16bd654f102" containerID="f8bc8a256291029dd7714e76e38300e1966377734122a9f915b11bd9b3e0f131" exitCode=0
Sep 10 23:01:41.035136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.035070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal" event={"ID":"3e2d8607ec87ae4c7192d16bd654f102","Type":"ContainerDied","Data":"f8bc8a256291029dd7714e76e38300e1966377734122a9f915b11bd9b3e0f131"}
Sep 10 23:01:41.036648 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.036619    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-whf9r" event={"ID":"07825aef-03b4-4691-8166-1cf8462cc3fc","Type":"ContainerStarted","Data":"1e901f5848f710a8e11e3ca2b482d63dd5ddd067a96a06f5185ca14609a12a3d"}
Sep 10 23:01:41.038124 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.038105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25" event={"ID":"2863dbdf-da97-48ce-8950-d0eeabdd9039","Type":"ContainerStarted","Data":"9c87fcdc43f0ee7c7b70f4a6e2860457f5d6f0cfea6db6fc1982f456a72f5c7a"}
Sep 10 23:01:41.039554 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.039532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n6rbk" event={"ID":"bc7dd59a-fbd1-49b2-b886-ee2429896ddc","Type":"ContainerStarted","Data":"1ea9de670d27bbe5ec4a2aa2ca7c5a60dff13b47c4923406cb14cd45b0bb0246"}
Sep 10 23:01:41.040925 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.040904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-246.ec2.internal" event={"ID":"8749419d048b9a7948ba1036740398d2","Type":"ContainerStarted","Data":"8c79dd73b2238cb61d2c4a38b621ae1711c5c52964dea512e2c345c2ef35c523"}
Sep 10 23:01:41.042393 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.042373    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b14fb8d-3467-4ca7-9200-d6ae4692aa22" containerID="084410ea98fdc16ec40f86a4d36769b444389dbfcee4e86f736052aa5bee308a" exitCode=0
Sep 10 23:01:41.042483 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.042435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md4pm" event={"ID":"0b14fb8d-3467-4ca7-9200-d6ae4692aa22","Type":"ContainerDied","Data":"084410ea98fdc16ec40f86a4d36769b444389dbfcee4e86f736052aa5bee308a"}
Sep 10 23:01:41.044074 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.044047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ctqkx" event={"ID":"21f78481-f131-45b0-ba1d-498e4f3d76bd","Type":"ContainerStarted","Data":"97f578014c22837d1f9af795ab4230e1e6383303345ccf0d86d9d68bf309e5ae"}
Sep 10 23:01:41.046657 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.046641    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:01:41.046980 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.046963    2563 generic.go:356] "Generic (PLEG): container finished" podID="df881283-5594-424d-98e4-e75fec3b7d19" containerID="68a033f2c9641fb3021bf5c1a31f3aadc9f22276e6bce711369e5082e55cecc7" exitCode=1
Sep 10 23:01:41.047033 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.046989    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks" event={"ID":"df881283-5594-424d-98e4-e75fec3b7d19","Type":"ContainerStarted","Data":"0d21e38306d19278a058a8572ef3788fb1d33562e3aac63d4420bde7df6e85a7"}
Sep 10 23:01:41.047033 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.047030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks" event={"ID":"df881283-5594-424d-98e4-e75fec3b7d19","Type":"ContainerStarted","Data":"39a860dcab5cf2abdecedd6fa003e90c38128b0e57bf30b9f59efe9ca517fdc7"}
Sep 10 23:01:41.047094 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.047059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks" event={"ID":"df881283-5594-424d-98e4-e75fec3b7d19","Type":"ContainerStarted","Data":"86df0efc5df932dfaba4873bf46d171d5b93d431cc2fd556bb2fec29cd400961"}
Sep 10 23:01:41.047094 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.047087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks" event={"ID":"df881283-5594-424d-98e4-e75fec3b7d19","Type":"ContainerStarted","Data":"de0a7f44d608f156bcd539887a5165fa49c09fb5ed24eb2b2f7cac230ef0570f"}
Sep 10 23:01:41.047153 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.047116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks" event={"ID":"df881283-5594-424d-98e4-e75fec3b7d19","Type":"ContainerDied","Data":"68a033f2c9641fb3021bf5c1a31f3aadc9f22276e6bce711369e5082e55cecc7"}
Sep 10 23:01:41.047153 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.047144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks" event={"ID":"df881283-5594-424d-98e4-e75fec3b7d19","Type":"ContainerStarted","Data":"0f8c80897eb5d7c8e0a8ff1f8275ad81a812bdd3b7aba94f0899ba245ecb03b2"}
Sep 10 23:01:41.077339 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.077257    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5j59t" podStartSLOduration=2.339052505 podStartE2EDuration="21.077236882s" totalImagesPullingTime="18.738184377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:41.05925867 +0000 UTC m=+21.750835853" watchObservedRunningTime="2026-09-10 23:01:41.077236882 +0000 UTC m=+21.768814064"
Sep 10 23:01:41.093423 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.093382    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-246.ec2.internal" podStartSLOduration=21.093365546 podStartE2EDuration="21.093365546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:41.092759272 +0000 UTC m=+21.784336477" watchObservedRunningTime="2026-09-10 23:01:41.093365546 +0000 UTC m=+21.784942732"
Sep 10 23:01:41.143699 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.143647    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n6rbk" podStartSLOduration=2.351607689 podStartE2EDuration="21.143617678s" totalImagesPullingTime="18.792009989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:41.143251614 +0000 UTC m=+21.834828797" watchObservedRunningTime="2026-09-10 23:01:41.143617678 +0000 UTC m=+21.835194862"
Sep 10 23:01:41.159477 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.159413    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-whf9r" podStartSLOduration=2.384216233 podStartE2EDuration="21.159398522s" totalImagesPullingTime="18.775182289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:41.158873576 +0000 UTC m=+21.850450782" watchObservedRunningTime="2026-09-10 23:01:41.159398522 +0000 UTC m=+21.850975705"
Sep 10 23:01:41.176910 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.176860    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ctqkx" podStartSLOduration=3.092387906 podStartE2EDuration="22.176844465s" totalImagesPullingTime="19.084456559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:41.176413004 +0000 UTC m=+21.867990187" watchObservedRunningTime="2026-09-10 23:01:41.176844465 +0000 UTC m=+21.868421649"
Sep 10 23:01:41.831338 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:41.831315    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 23:01:42.051136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:42.051046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pgsk2" event={"ID":"246cb358-713a-4429-aac2-8853a3d0dbdd","Type":"ContainerStarted","Data":"9e8c422f3169b5db794df5c5429bd5db504b12ba2abdb31bd8efeeed67b72af2"}
Sep 10 23:01:42.053157 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:42.053101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vjp6c" event={"ID":"f5b7c54d-0f06-47e3-b7c8-0d6464f7f9b7","Type":"ContainerStarted","Data":"405cd3ceab380f6d027d6d8076b0b709ba022bb30ea750c60ee657df8dc42116"}
Sep 10 23:01:42.055098 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:42.055068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal" event={"ID":"3e2d8607ec87ae4c7192d16bd654f102","Type":"ContainerStarted","Data":"9b051c540252d9b732cd836c3677e3d74dcddcff142843d0c32abb8fe0fd3fb2"}
Sep 10 23:01:42.057104 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:42.057079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25" event={"ID":"2863dbdf-da97-48ce-8950-d0eeabdd9039","Type":"ContainerStarted","Data":"d8a6bf06c08714bee01739570495d64a76641f72da9f33fd9305a3b2d35b837a"}
Sep 10 23:01:42.070925 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:42.070868    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pgsk2" podStartSLOduration=1.9155289720000002 podStartE2EDuration="22.070847846s" totalImagesPullingTime="20.155318874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-10 23:01:42.070560004 +0000 UTC m=+22.762137233" watchObservedRunningTime="2026-09-10 23:01:42.070847846 +0000 UTC m=+22.762425031"
Sep 10 23:01:42.085147 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:42.085089    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vjp6c" podStartSLOduration=3.360244219 podStartE2EDuration="22.085067665s" totalImagesPullingTime="18.724823446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:42.084722371 +0000 UTC m=+22.776299582" watchObservedRunningTime="2026-09-10 23:01:42.085067665 +0000 UTC m=+22.776644853"
Sep 10 23:01:42.099337 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:42.099280    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-246.ec2.internal" podStartSLOduration=22.099259509 podStartE2EDuration="22.099259509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:42.099235428 +0000 UTC m=+22.790812637" watchObservedRunningTime="2026-09-10 23:01:42.099259509 +0000 UTC m=+22.790836691"
Sep 10 23:01:42.775368 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:42.774865    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T23:01:41.83133445Z","UUID":"ff523ed5-1d79-4f7f-a0cb-0539d49bed42","Handler":null,"Name":"","Endpoint":""}
Sep 10 23:01:42.778108 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:42.777625    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 23:01:42.778108 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:42.777661    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 23:01:42.808472 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:42.808413    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:42.808472 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:42.808439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:42.808472 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:42.808454    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:42.808727 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:42.808570    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:42.808791 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:42.808764    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:42.808875 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:42.808845    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:43.065296 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:43.065266    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:01:43.065780 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:43.065750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks" event={"ID":"df881283-5594-424d-98e4-e75fec3b7d19","Type":"ContainerStarted","Data":"b423c4f7d19802c286b39dbab5bd525f519dbecb736b4f5aae0d427dedefe4a7"}
Sep 10 23:01:43.068464 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:43.068375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25" event={"ID":"2863dbdf-da97-48ce-8950-d0eeabdd9039","Type":"ContainerStarted","Data":"5c5b2ee7405b6dbabcd5a79d314d318c77fd38e39d9cd5e5f6e9c7a6a23fcd76"}
Sep 10 23:01:43.085684 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:43.085629    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7kg25" podStartSLOduration=2.266610339 podStartE2EDuration="23.085611009s" totalImagesPullingTime="20.81900067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:43.084430974 +0000 UTC m=+23.776008179" watchObservedRunningTime="2026-09-10 23:01:43.085611009 +0000 UTC m=+23.777188193"
Sep 10 23:01:44.808367 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:44.808331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:44.808800 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:44.808331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:44.808800 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:44.808438    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:44.808800 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:44.808549    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:44.808800 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:44.808339    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:44.808800 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:44.808660    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:45.992386 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:45.992339    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pgsk2"
Sep 10 23:01:45.993063 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:45.993036    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pgsk2"
Sep 10 23:01:46.807791 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:46.807611    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:46.807946 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:46.807612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:46.807946 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:46.807884    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:46.807946 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:46.807612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:46.808043 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:46.808009    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:46.808043 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:46.807939    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:47.080737 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:47.080661    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:01:47.081205 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:47.081047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks" event={"ID":"df881283-5594-424d-98e4-e75fec3b7d19","Type":"ContainerStarted","Data":"f3eee087c113861554779073d37cc7afffdacf8db5171b412db243221506e9cd"}
Sep 10 23:01:47.081392 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:47.081365    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:47.081531 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:47.081513    2563 scope.go:117] "RemoveContainer" containerID="68a033f2c9641fb3021bf5c1a31f3aadc9f22276e6bce711369e5082e55cecc7"
Sep 10 23:01:47.083116 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:47.083093    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b14fb8d-3467-4ca7-9200-d6ae4692aa22" containerID="89c13f368133c42b88d789fca6c86bc6c71ba0ab718444282388987f7b9ca859" exitCode=0
Sep 10 23:01:47.083221 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:47.083130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md4pm" event={"ID":"0b14fb8d-3467-4ca7-9200-d6ae4692aa22","Type":"ContainerDied","Data":"89c13f368133c42b88d789fca6c86bc6c71ba0ab718444282388987f7b9ca859"}
Sep 10 23:01:47.097975 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:47.097939    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:48.088703 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:48.088680    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:01:48.089302 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:48.089003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks" event={"ID":"df881283-5594-424d-98e4-e75fec3b7d19","Type":"ContainerStarted","Data":"ba3382fb637be054a161891614736113bf1392bd03f62af5d582cdb27b44418b"}
Sep 10 23:01:48.089302 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:48.089174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:48.089302 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:48.089223    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:48.104399 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:48.104373    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:01:48.120193 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:48.120142    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks" podStartSLOduration=9.364475107 podStartE2EDuration="28.120127547s" totalImagesPullingTime="18.75565244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:48.11964012 +0000 UTC m=+28.811217315" watchObservedRunningTime="2026-09-10 23:01:48.120127547 +0000 UTC m=+28.811704731"
Sep 10 23:01:48.249486 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:48.249461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xdms4"]
Sep 10 23:01:48.249640 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:48.249625    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:48.249758 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:48.249735    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:48.252783 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:48.252759    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kr684"]
Sep 10 23:01:48.253104 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:48.252871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:48.253104 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:48.252965    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:48.255846 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:48.255821    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v8sx8"]
Sep 10 23:01:48.255962 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:48.255949    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:48.256091 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:48.256066    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:49.027511 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:49.027462    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pgsk2"
Sep 10 23:01:49.027686 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:49.027601    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 23:01:49.028261 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:49.028232    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pgsk2"
Sep 10 23:01:49.094113 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:49.094082    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b14fb8d-3467-4ca7-9200-d6ae4692aa22" containerID="0bc3d4b8b2b79cb5712fc843c7ae045cb9d8159f24eabdf42e6b3d2a940c72c6" exitCode=0
Sep 10 23:01:49.094705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:49.094163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md4pm" event={"ID":"0b14fb8d-3467-4ca7-9200-d6ae4692aa22","Type":"ContainerDied","Data":"0bc3d4b8b2b79cb5712fc843c7ae045cb9d8159f24eabdf42e6b3d2a940c72c6"}
Sep 10 23:01:49.809373 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:49.809197    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:49.809566 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:49.809262    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:49.809566 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:49.809279    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:49.809566 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:49.809437    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:49.809680 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:49.809635    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:49.809752 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:49.809728    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:51.101228 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:51.101188    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b14fb8d-3467-4ca7-9200-d6ae4692aa22" containerID="971218a8344632fc73ce634b1e76ac50bb96e01aae0d4640e5ff8981b2fd9845" exitCode=0
Sep 10 23:01:51.101702 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:51.101250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md4pm" event={"ID":"0b14fb8d-3467-4ca7-9200-d6ae4692aa22","Type":"ContainerDied","Data":"971218a8344632fc73ce634b1e76ac50bb96e01aae0d4640e5ff8981b2fd9845"}
Sep 10 23:01:51.807635 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:51.807598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:51.807818 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:51.807653    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:51.807818 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:51.807598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:51.807818 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:51.807761    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v8sx8" podUID="54f0cb94-e7d5-4b86-8602-dd8b46138dab"
Sep 10 23:01:51.808003 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:51.807835    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr684" podUID="34885b0c-c971-4056-a091-1764a294ea66"
Sep 10 23:01:51.808003 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:51.807918    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdms4" podUID="ab0193de-c9ef-4b5c-a865-bf880e0761d0"
Sep 10 23:01:52.554922 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:52.554887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:52.555393 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:52.554933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpth\" (UniqueName: \"kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth\") pod \"network-check-target-kr684\" (UID: \"34885b0c-c971-4056-a091-1764a294ea66\") " pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:52.555393 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:52.554964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs\") pod \"network-metrics-daemon-v8sx8\" (UID: \"54f0cb94-e7d5-4b86-8602-dd8b46138dab\") " pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:52.555393 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:52.555067    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:52.555393 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:52.555086    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 23:01:52.555393 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:52.555113    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 23:01:52.555393 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:52.555127    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5mpth for pod openshift-network-diagnostics/network-check-target-kr684: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:52.555393 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:52.555133    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs podName:54f0cb94-e7d5-4b86-8602-dd8b46138dab nodeName:}" failed. No retries permitted until 2026-09-10 23:02:24.5551155 +0000 UTC m=+65.246692685 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs") pod "network-metrics-daemon-v8sx8" (UID: "54f0cb94-e7d5-4b86-8602-dd8b46138dab") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 23:01:52.555393 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:52.555067    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:52.555393 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:52.555184    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth podName:34885b0c-c971-4056-a091-1764a294ea66 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:24.555166915 +0000 UTC m=+65.246744077 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5mpth" (UniqueName: "kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth") pod "network-check-target-kr684" (UID: "34885b0c-c971-4056-a091-1764a294ea66") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 23:01:52.555393 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:52.555204    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret podName:ab0193de-c9ef-4b5c-a865-bf880e0761d0 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:24.555194574 +0000 UTC m=+65.246771736 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret") pod "global-pull-secret-syncer-xdms4" (UID: "ab0193de-c9ef-4b5c-a865-bf880e0761d0") : object "kube-system"/"original-pull-secret" not registered
Sep 10 23:01:53.138030 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.137948    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-246.ec2.internal" event="NodeReady"
Sep 10 23:01:53.138199 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.138101    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 23:01:53.176468 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.176430    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-666f85fc79-xmk8h"]
Sep 10 23:01:53.196604 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.196485    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qq99b"]
Sep 10 23:01:53.196760 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.196620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.200429 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.199671    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 23:01:53.200429 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.199980    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 23:01:53.200429 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.200190    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-78nrb\""
Sep 10 23:01:53.200717 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.200520    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 23:01:53.212446 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.212420    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666f85fc79-xmk8h"]
Sep 10 23:01:53.212609 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.212463    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hx99c"]
Sep 10 23:01:53.212609 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.212464    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 23:01:53.212609 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.212558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:53.214683 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.214661    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 23:01:53.214798 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.214689    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-497ss\""
Sep 10 23:01:53.214857 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.214831    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 23:01:53.234921 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.234854    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qq99b"]
Sep 10 23:01:53.234921 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.234926    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hx99c"]
Sep 10 23:01:53.235484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.234980    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hx99c"
Sep 10 23:01:53.237317 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.237288    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 23:01:53.237477 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.237338    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 23:01:53.237477 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.237395    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tqzlq\""
Sep 10 23:01:53.237638 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.237617    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 23:01:53.259962 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.259932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d44f82f8-998f-47b3-8470-a6a8e1c66a98-ca-trust-extracted\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.259962 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.259963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-bound-sa-token\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.260213 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.259987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d44f82f8-998f-47b3-8470-a6a8e1c66a98-installation-pull-secrets\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.260213 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.260022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjwpf\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-kube-api-access-tjwpf\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.260213 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.260071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-certificates\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.260213 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.260191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d44f82f8-998f-47b3-8470-a6a8e1c66a98-trusted-ca\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.260395 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.260240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d44f82f8-998f-47b3-8470-a6a8e1c66a98-image-registry-private-configuration\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.260395 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.260267    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.342412 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.342354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgsk2_246cb358-713a-4429-aac2-8853a3d0dbdd/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:53.361214 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-879xk\" (UniqueName: \"kubernetes.io/projected/43c78e11-d941-4f95-99b2-abd00726363e-kube-api-access-879xk\") pod \"ingress-canary-hx99c\" (UID: \"43c78e11-d941-4f95-99b2-abd00726363e\") " pod="openshift-ingress-canary/ingress-canary-hx99c"
Sep 10 23:01:53.361393 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d44f82f8-998f-47b3-8470-a6a8e1c66a98-image-registry-private-configuration\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.361393 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.361393 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d44f82f8-998f-47b3-8470-a6a8e1c66a98-ca-trust-extracted\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.361393 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-bound-sa-token\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.361393 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwtv6\" (UniqueName: \"kubernetes.io/projected/d3aff66a-de8c-4842-902b-fe0015f1ae4b-kube-api-access-fwtv6\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:53.361652 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361372    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43c78e11-d941-4f95-99b2-abd00726363e-cert\") pod \"ingress-canary-hx99c\" (UID: \"43c78e11-d941-4f95-99b2-abd00726363e\") " pod="openshift-ingress-canary/ingress-canary-hx99c"
Sep 10 23:01:53.361652 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d44f82f8-998f-47b3-8470-a6a8e1c66a98-installation-pull-secrets\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.361652 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:53.361429    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:01:53.361652 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:53.361523    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666f85fc79-xmk8h: secret "image-registry-tls" not found
Sep 10 23:01:53.361652 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjwpf\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-kube-api-access-tjwpf\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.361652 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3aff66a-de8c-4842-902b-fe0015f1ae4b-config-volume\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:53.361652 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3aff66a-de8c-4842-902b-fe0015f1ae4b-tmp-dir\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:53.361976 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-certificates\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.361976 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:53.361689    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls podName:d44f82f8-998f-47b3-8470-a6a8e1c66a98 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:53.861666902 +0000 UTC m=+34.553244073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls") pod "image-registry-666f85fc79-xmk8h" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98") : secret "image-registry-tls" not found
Sep 10 23:01:53.361976 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:53.361976 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d44f82f8-998f-47b3-8470-a6a8e1c66a98-trusted-ca\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.361976 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.361809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d44f82f8-998f-47b3-8470-a6a8e1c66a98-ca-trust-extracted\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.362355 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.362184    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-certificates\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.362750 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.362724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d44f82f8-998f-47b3-8470-a6a8e1c66a98-trusted-ca\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.365961 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.365942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d44f82f8-998f-47b3-8470-a6a8e1c66a98-installation-pull-secrets\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.366065 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.365983    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d44f82f8-998f-47b3-8470-a6a8e1c66a98-image-registry-private-configuration\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.375919 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.375884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-bound-sa-token\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.376062 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.376011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjwpf\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-kube-api-access-tjwpf\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.424927 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.424841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pgsk2_246cb358-713a-4429-aac2-8853a3d0dbdd/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:53.462747 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.462714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-879xk\" (UniqueName: \"kubernetes.io/projected/43c78e11-d941-4f95-99b2-abd00726363e-kube-api-access-879xk\") pod \"ingress-canary-hx99c\" (UID: \"43c78e11-d941-4f95-99b2-abd00726363e\") " pod="openshift-ingress-canary/ingress-canary-hx99c"
Sep 10 23:01:53.462929 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.462783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwtv6\" (UniqueName: \"kubernetes.io/projected/d3aff66a-de8c-4842-902b-fe0015f1ae4b-kube-api-access-fwtv6\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:53.462929 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.462808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43c78e11-d941-4f95-99b2-abd00726363e-cert\") pod \"ingress-canary-hx99c\" (UID: \"43c78e11-d941-4f95-99b2-abd00726363e\") " pod="openshift-ingress-canary/ingress-canary-hx99c"
Sep 10 23:01:53.463043 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.462970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3aff66a-de8c-4842-902b-fe0015f1ae4b-config-volume\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:53.463043 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.463013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3aff66a-de8c-4842-902b-fe0015f1ae4b-tmp-dir\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:53.463159 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.463049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:53.463220 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:53.463154    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:01:53.463268 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:53.463225    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls podName:d3aff66a-de8c-4842-902b-fe0015f1ae4b nodeName:}" failed. No retries permitted until 2026-09-10 23:01:53.963203639 +0000 UTC m=+34.654780818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls") pod "dns-default-qq99b" (UID: "d3aff66a-de8c-4842-902b-fe0015f1ae4b") : secret "dns-default-metrics-tls" not found
Sep 10 23:01:53.463364 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.463342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3aff66a-de8c-4842-902b-fe0015f1ae4b-tmp-dir\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:53.463569 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.463547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3aff66a-de8c-4842-902b-fe0015f1ae4b-config-volume\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:53.465575 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.465550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43c78e11-d941-4f95-99b2-abd00726363e-cert\") pod \"ingress-canary-hx99c\" (UID: \"43c78e11-d941-4f95-99b2-abd00726363e\") " pod="openshift-ingress-canary/ingress-canary-hx99c"
Sep 10 23:01:53.472081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.472057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwtv6\" (UniqueName: \"kubernetes.io/projected/d3aff66a-de8c-4842-902b-fe0015f1ae4b-kube-api-access-fwtv6\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:53.472219 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.472085    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-879xk\" (UniqueName: \"kubernetes.io/projected/43c78e11-d941-4f95-99b2-abd00726363e-kube-api-access-879xk\") pod \"ingress-canary-hx99c\" (UID: \"43c78e11-d941-4f95-99b2-abd00726363e\") " pod="openshift-ingress-canary/ingress-canary-hx99c"
Sep 10 23:01:53.545353 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.545317    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hx99c"
Sep 10 23:01:53.812535 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.812488    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:01:53.812973 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.812639    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:01:53.812973 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.812723    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:01:53.815477 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.815258    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 23:01:53.815477 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.815278    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gcwbm\""
Sep 10 23:01:53.815477 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.815311    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 23:01:53.815477 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.815262    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 23:01:53.815839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.815565    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 23:01:53.815839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.815706    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-glltp\""
Sep 10 23:01:53.855755 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.855726    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hx99c"]
Sep 10 23:01:53.860718 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:01:53.860687    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c78e11_d941_4f95_99b2_abd00726363e.slice/crio-f58c460c70c35632bb7ac9cc1396905f270939aadb1be13154b251a8f0ac7366 WatchSource:0}: Error finding container f58c460c70c35632bb7ac9cc1396905f270939aadb1be13154b251a8f0ac7366: Status 404 returned error can't find the container with id f58c460c70c35632bb7ac9cc1396905f270939aadb1be13154b251a8f0ac7366
Sep 10 23:01:53.865695 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.865672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:53.865841 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:53.865824    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:01:53.865879 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:53.865842    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666f85fc79-xmk8h: secret "image-registry-tls" not found
Sep 10 23:01:53.865918 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:53.865896    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls podName:d44f82f8-998f-47b3-8470-a6a8e1c66a98 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:54.865875573 +0000 UTC m=+35.557452737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls") pod "image-registry-666f85fc79-xmk8h" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98") : secret "image-registry-tls" not found
Sep 10 23:01:53.966297 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:53.966266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:53.966469 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:53.966406    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:01:53.966551 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:53.966485    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls podName:d3aff66a-de8c-4842-902b-fe0015f1ae4b nodeName:}" failed. No retries permitted until 2026-09-10 23:01:54.966472947 +0000 UTC m=+35.658050109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls") pod "dns-default-qq99b" (UID: "d3aff66a-de8c-4842-902b-fe0015f1ae4b") : secret "dns-default-metrics-tls" not found
Sep 10 23:01:54.110461 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:54.110382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hx99c" event={"ID":"43c78e11-d941-4f95-99b2-abd00726363e","Type":"ContainerStarted","Data":"f58c460c70c35632bb7ac9cc1396905f270939aadb1be13154b251a8f0ac7366"}
Sep 10 23:01:54.874873 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:54.874830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:54.875439 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:54.875017    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:01:54.875439 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:54.875044    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666f85fc79-xmk8h: secret "image-registry-tls" not found
Sep 10 23:01:54.875439 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:54.875115    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls podName:d44f82f8-998f-47b3-8470-a6a8e1c66a98 nodeName:}" failed. No retries permitted until 2026-09-10 23:01:56.87509505 +0000 UTC m=+37.566672228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls") pod "image-registry-666f85fc79-xmk8h" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98") : secret "image-registry-tls" not found
Sep 10 23:01:54.975585 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:54.975542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:54.975782 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:54.975706    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:01:54.975782 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:54.975774    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls podName:d3aff66a-de8c-4842-902b-fe0015f1ae4b nodeName:}" failed. No retries permitted until 2026-09-10 23:01:56.975755267 +0000 UTC m=+37.667332438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls") pod "dns-default-qq99b" (UID: "d3aff66a-de8c-4842-902b-fe0015f1ae4b") : secret "dns-default-metrics-tls" not found
Sep 10 23:01:56.119828 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:56.119738    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b14fb8d-3467-4ca7-9200-d6ae4692aa22" containerID="71c4903989c8257952853eb7b441f13069e4f963e0581f357ac701f956e05579" exitCode=0
Sep 10 23:01:56.119828 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:56.119790    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md4pm" event={"ID":"0b14fb8d-3467-4ca7-9200-d6ae4692aa22","Type":"ContainerDied","Data":"71c4903989c8257952853eb7b441f13069e4f963e0581f357ac701f956e05579"}
Sep 10 23:01:56.894864 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:56.894819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:01:56.895019 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:56.894964    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:01:56.895019 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:56.894984    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666f85fc79-xmk8h: secret "image-registry-tls" not found
Sep 10 23:01:56.895090 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:56.895038    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls podName:d44f82f8-998f-47b3-8470-a6a8e1c66a98 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:00.895022958 +0000 UTC m=+41.586600124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls") pod "image-registry-666f85fc79-xmk8h" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98") : secret "image-registry-tls" not found
Sep 10 23:01:56.995698 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:56.995654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:01:56.995868 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:56.995781    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:01:56.995868 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:01:56.995837    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls podName:d3aff66a-de8c-4842-902b-fe0015f1ae4b nodeName:}" failed. No retries permitted until 2026-09-10 23:02:00.995822092 +0000 UTC m=+41.687399253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls") pod "dns-default-qq99b" (UID: "d3aff66a-de8c-4842-902b-fe0015f1ae4b") : secret "dns-default-metrics-tls" not found
Sep 10 23:01:57.126868 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:57.126781    2563 generic.go:356] "Generic (PLEG): container finished" podID="0b14fb8d-3467-4ca7-9200-d6ae4692aa22" containerID="293f9d54089679bbeb04eccb96e109786fa9fcc06f6455190810d804085ce835" exitCode=0
Sep 10 23:01:57.126868 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:57.126831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md4pm" event={"ID":"0b14fb8d-3467-4ca7-9200-d6ae4692aa22","Type":"ContainerDied","Data":"293f9d54089679bbeb04eccb96e109786fa9fcc06f6455190810d804085ce835"}
Sep 10 23:01:58.132812 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:58.132778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-md4pm" event={"ID":"0b14fb8d-3467-4ca7-9200-d6ae4692aa22","Type":"ContainerStarted","Data":"393b27dbe71e78be0c6d7305fce6267228399b8e189682a71eeab6c6656afb19"}
Sep 10 23:01:58.154292 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:58.154233    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-md4pm" podStartSLOduration=7.975674824 podStartE2EDuration="39.15421502s" totalImagesPullingTime="31.178540196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:58.152930962 +0000 UTC m=+38.844508146" watchObservedRunningTime="2026-09-10 23:01:58.15421502 +0000 UTC m=+38.845792201"
Sep 10 23:01:59.136697 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:59.136653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hx99c" event={"ID":"43c78e11-d941-4f95-99b2-abd00726363e","Type":"ContainerStarted","Data":"b62aa7de909067cf23581407445d03d08b474403fd50ba90c091e4531ae75f73"}
Sep 10 23:01:59.153818 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:01:59.153767    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hx99c" podStartSLOduration=1.801531016 podStartE2EDuration="6.153749537s" totalImagesPullingTime="4.352218521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:01:59.152454412 +0000 UTC m=+39.844031615" watchObservedRunningTime="2026-09-10 23:01:59.153749537 +0000 UTC m=+39.845326720"
Sep 10 23:02:00.931071 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:00.931034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:02:00.931443 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:00.931197    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:02:00.931443 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:00.931221    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666f85fc79-xmk8h: secret "image-registry-tls" not found
Sep 10 23:02:00.931443 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:00.931276    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls podName:d44f82f8-998f-47b3-8470-a6a8e1c66a98 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:08.931261932 +0000 UTC m=+49.622839093 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls") pod "image-registry-666f85fc79-xmk8h" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98") : secret "image-registry-tls" not found
Sep 10 23:02:01.032144 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:01.032102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:02:01.032294 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:01.032213    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:02:01.032294 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:01.032280    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls podName:d3aff66a-de8c-4842-902b-fe0015f1ae4b nodeName:}" failed. No retries permitted until 2026-09-10 23:02:09.032267046 +0000 UTC m=+49.723844213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls") pod "dns-default-qq99b" (UID: "d3aff66a-de8c-4842-902b-fe0015f1ae4b") : secret "dns-default-metrics-tls" not found
Sep 10 23:02:05.788968 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.788931    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"]
Sep 10 23:02:05.842210 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.842172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:05.845091 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.845066    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 10 23:02:05.845768 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.845743    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"]
Sep 10 23:02:05.847371 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.847350    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 23:02:05.847476 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.847350    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 10 23:02:05.847476 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.847462    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8mhxt\""
Sep 10 23:02:05.847618 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.847352    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 23:02:05.885808 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.885777    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-96fls"]
Sep 10 23:02:05.907612 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.907580    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8484fd4884-9th5k"]
Sep 10 23:02:05.907756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.907738    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:05.910036 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.910013    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 10 23:02:05.910164 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.910115    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 23:02:05.910385 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.910366    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 10 23:02:05.910635 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.910617    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8ccd5\""
Sep 10 23:02:05.910704 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.910641    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 23:02:05.915392 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.915374    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 10 23:02:05.931753 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.931717    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-96fls"]
Sep 10 23:02:05.931753 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.931759    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8484fd4884-9th5k"]
Sep 10 23:02:05.931967 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.931865    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:05.934517 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.934478    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 10 23:02:05.934635 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.934609    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 10 23:02:05.934794 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.934778    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 23:02:05.934830 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.934816    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 10 23:02:05.935029 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.935015    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 10 23:02:05.935077 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.935064    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zkt9j\""
Sep 10 23:02:05.935267 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.935254    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 23:02:05.967007 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.966967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:05.967007 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.967013    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc9zh\" (UniqueName: \"kubernetes.io/projected/d7a69d1d-f105-49eb-ad0e-808c10d9adae-kube-api-access-bc9zh\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:05.967262 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:05.967083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7a69d1d-f105-49eb-ad0e-808c10d9adae-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:06.067764 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.067727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9zh\" (UniqueName: \"kubernetes.io/projected/d7a69d1d-f105-49eb-ad0e-808c10d9adae-kube-api-access-bc9zh\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:06.067952 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.067780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7a69d1d-f105-49eb-ad0e-808c10d9adae-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:06.067952 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.067809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-default-certificate\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.067952 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.067848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/77ac1f77-479a-4611-b90c-c5ffb2242a9f-serving-cert\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.067952 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.067882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-stats-auth\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.068094 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.067956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77ac1f77-479a-4611-b90c-c5ffb2242a9f-tmp\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.068094 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.067993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/77ac1f77-479a-4611-b90c-c5ffb2242a9f-snapshots\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.068094 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.068040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6k89\" (UniqueName: \"kubernetes.io/projected/810a3f8a-76ae-4456-82d1-2f50e10721ac-kube-api-access-c6k89\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.068094 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.068071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77ac1f77-479a-4611-b90c-c5ffb2242a9f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.068211 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.068096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.068211 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.068121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77ac1f77-479a-4611-b90c-c5ffb2242a9f-service-ca-bundle\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.068211 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.068141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx28q\" (UniqueName: \"kubernetes.io/projected/77ac1f77-479a-4611-b90c-c5ffb2242a9f-kube-api-access-hx28q\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.068211 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.068185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:06.068211 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.068201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.068353 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:06.068298    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 23:02:06.068385 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:06.068373    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls podName:d7a69d1d-f105-49eb-ad0e-808c10d9adae nodeName:}" failed. No retries permitted until 2026-09-10 23:02:06.568359137 +0000 UTC m=+47.259936298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bwkkv" (UID: "d7a69d1d-f105-49eb-ad0e-808c10d9adae") : secret "cluster-monitoring-operator-tls" not found
Sep 10 23:02:06.068620 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.068602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d7a69d1d-f105-49eb-ad0e-808c10d9adae-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:06.079208 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.079179    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc9zh\" (UniqueName: \"kubernetes.io/projected/d7a69d1d-f105-49eb-ad0e-808c10d9adae-kube-api-access-bc9zh\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:06.169386 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.169339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-default-certificate\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.169597 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.169402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/77ac1f77-479a-4611-b90c-c5ffb2242a9f-serving-cert\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.169597 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.169430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-stats-auth\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.169597 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.169453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77ac1f77-479a-4611-b90c-c5ffb2242a9f-tmp\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.169597 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.169467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/77ac1f77-479a-4611-b90c-c5ffb2242a9f-snapshots\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.169597 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.169483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6k89\" (UniqueName: \"kubernetes.io/projected/810a3f8a-76ae-4456-82d1-2f50e10721ac-kube-api-access-c6k89\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.169597 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.169526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77ac1f77-479a-4611-b90c-c5ffb2242a9f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.169597 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.169548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.169597 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.169582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77ac1f77-479a-4611-b90c-c5ffb2242a9f-service-ca-bundle\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.170012 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.169613    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx28q\" (UniqueName: \"kubernetes.io/projected/77ac1f77-479a-4611-b90c-c5ffb2242a9f-kube-api-access-hx28q\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.170012 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.169669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.170012 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:06.169815    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle podName:810a3f8a-76ae-4456-82d1-2f50e10721ac nodeName:}" failed. No retries permitted until 2026-09-10 23:02:06.669795859 +0000 UTC m=+47.361373022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle") pod "router-default-8484fd4884-9th5k" (UID: "810a3f8a-76ae-4456-82d1-2f50e10721ac") : configmap references non-existent config key: service-ca.crt
Sep 10 23:02:06.170012 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:06.169914    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:02:06.170012 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:06.169996    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs podName:810a3f8a-76ae-4456-82d1-2f50e10721ac nodeName:}" failed. No retries permitted until 2026-09-10 23:02:06.669973176 +0000 UTC m=+47.361550357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs") pod "router-default-8484fd4884-9th5k" (UID: "810a3f8a-76ae-4456-82d1-2f50e10721ac") : secret "router-metrics-certs-default" not found
Sep 10 23:02:06.170262 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.170177    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77ac1f77-479a-4611-b90c-c5ffb2242a9f-tmp\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.170321 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.170253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/77ac1f77-479a-4611-b90c-c5ffb2242a9f-snapshots\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.170551 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.170526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77ac1f77-479a-4611-b90c-c5ffb2242a9f-service-ca-bundle\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.171209 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.171182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77ac1f77-479a-4611-b90c-c5ffb2242a9f-trusted-ca-bundle\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.172054 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.172025    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-stats-auth\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.172142 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.172120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/77ac1f77-479a-4611-b90c-c5ffb2242a9f-serving-cert\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.172257 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.172241    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-default-certificate\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.177660 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.177639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx28q\" (UniqueName: \"kubernetes.io/projected/77ac1f77-479a-4611-b90c-c5ffb2242a9f-kube-api-access-hx28q\") pod \"insights-operator-5844b856db-96fls\" (UID: \"77ac1f77-479a-4611-b90c-c5ffb2242a9f\") " pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.177755 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.177739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6k89\" (UniqueName: \"kubernetes.io/projected/810a3f8a-76ae-4456-82d1-2f50e10721ac-kube-api-access-c6k89\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.216335 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.216298    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-96fls"
Sep 10 23:02:06.511953 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.511871    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-96fls"]
Sep 10 23:02:06.514860 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:02:06.514832    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ac1f77_479a_4611_b90c_c5ffb2242a9f.slice/crio-123da132a5f2d1935d15b7a0f560aadc981f712100c00bb3595039ee877842a4 WatchSource:0}: Error finding container 123da132a5f2d1935d15b7a0f560aadc981f712100c00bb3595039ee877842a4: Status 404 returned error can't find the container with id 123da132a5f2d1935d15b7a0f560aadc981f712100c00bb3595039ee877842a4
Sep 10 23:02:06.574408 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.574364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:06.574614 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:06.574538    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 23:02:06.574614 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:06.574609    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls podName:d7a69d1d-f105-49eb-ad0e-808c10d9adae nodeName:}" failed. No retries permitted until 2026-09-10 23:02:07.57459229 +0000 UTC m=+48.266169452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bwkkv" (UID: "d7a69d1d-f105-49eb-ad0e-808c10d9adae") : secret "cluster-monitoring-operator-tls" not found
Sep 10 23:02:06.675578 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.675544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.675782 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:06.675721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:06.675782 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:06.675751    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle podName:810a3f8a-76ae-4456-82d1-2f50e10721ac nodeName:}" failed. No retries permitted until 2026-09-10 23:02:07.675731635 +0000 UTC m=+48.367308811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle") pod "router-default-8484fd4884-9th5k" (UID: "810a3f8a-76ae-4456-82d1-2f50e10721ac") : configmap references non-existent config key: service-ca.crt
Sep 10 23:02:06.675907 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:06.675805    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:02:06.675907 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:06.675854    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs podName:810a3f8a-76ae-4456-82d1-2f50e10721ac nodeName:}" failed. No retries permitted until 2026-09-10 23:02:07.675843798 +0000 UTC m=+48.367420960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs") pod "router-default-8484fd4884-9th5k" (UID: "810a3f8a-76ae-4456-82d1-2f50e10721ac") : secret "router-metrics-certs-default" not found
Sep 10 23:02:07.155292 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:07.155261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-96fls" event={"ID":"77ac1f77-479a-4611-b90c-c5ffb2242a9f","Type":"ContainerStarted","Data":"123da132a5f2d1935d15b7a0f560aadc981f712100c00bb3595039ee877842a4"}
Sep 10 23:02:07.584610 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:07.584571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:07.584799 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:07.584742    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 23:02:07.584841 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:07.584824    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls podName:d7a69d1d-f105-49eb-ad0e-808c10d9adae nodeName:}" failed. No retries permitted until 2026-09-10 23:02:09.584804082 +0000 UTC m=+50.276381250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bwkkv" (UID: "d7a69d1d-f105-49eb-ad0e-808c10d9adae") : secret "cluster-monitoring-operator-tls" not found
Sep 10 23:02:07.685077 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:07.685040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:07.685249 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:07.685108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:07.685249 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:07.685222    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:02:07.685334 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:07.685257    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle podName:810a3f8a-76ae-4456-82d1-2f50e10721ac nodeName:}" failed. No retries permitted until 2026-09-10 23:02:09.685242385 +0000 UTC m=+50.376819567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle") pod "router-default-8484fd4884-9th5k" (UID: "810a3f8a-76ae-4456-82d1-2f50e10721ac") : configmap references non-existent config key: service-ca.crt
Sep 10 23:02:07.685334 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:07.685280    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs podName:810a3f8a-76ae-4456-82d1-2f50e10721ac nodeName:}" failed. No retries permitted until 2026-09-10 23:02:09.685265503 +0000 UTC m=+50.376842665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs") pod "router-default-8484fd4884-9th5k" (UID: "810a3f8a-76ae-4456-82d1-2f50e10721ac") : secret "router-metrics-certs-default" not found
Sep 10 23:02:08.997949 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:08.997914    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:02:08.998311 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:08.998034    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:02:08.998311 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:08.998048    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666f85fc79-xmk8h: secret "image-registry-tls" not found
Sep 10 23:02:08.998311 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:08.998099    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls podName:d44f82f8-998f-47b3-8470-a6a8e1c66a98 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:24.998085913 +0000 UTC m=+65.689663074 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls") pod "image-registry-666f85fc79-xmk8h" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98") : secret "image-registry-tls" not found
Sep 10 23:02:09.098681 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.098647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:02:09.098834 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:09.098807    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:02:09.098873 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:09.098868    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls podName:d3aff66a-de8c-4842-902b-fe0015f1ae4b nodeName:}" failed. No retries permitted until 2026-09-10 23:02:25.098850173 +0000 UTC m=+65.790427356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls") pod "dns-default-qq99b" (UID: "d3aff66a-de8c-4842-902b-fe0015f1ae4b") : secret "dns-default-metrics-tls" not found
Sep 10 23:02:09.162306 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.162272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-96fls" event={"ID":"77ac1f77-479a-4611-b90c-c5ffb2242a9f","Type":"ContainerStarted","Data":"943bb2d652844a5acfa835e658b48dfb9ac880e910641c42e0fa06f98bb73cd8"}
Sep 10 23:02:09.179427 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.179384    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-96fls" podStartSLOduration=2.210045588 podStartE2EDuration="4.179368406s" totalImagesPullingTime="1.969322818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:09.178685461 +0000 UTC m=+49.870262664" watchObservedRunningTime="2026-09-10 23:02:09.179368406 +0000 UTC m=+49.870945583"
Sep 10 23:02:09.497101 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.497060    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g8zrr"]
Sep 10 23:02:09.500457 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.500429    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:09.503042 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.503023    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 23:02:09.503159 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.503059    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 23:02:09.503159 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.503070    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w4w8p\""
Sep 10 23:02:09.510886 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.510864    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g8zrr"]
Sep 10 23:02:09.602993 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.602960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:09.602993 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.603000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmk7\" (UniqueName: \"kubernetes.io/projected/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-kube-api-access-lgmk7\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:09.603231 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.603021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:09.603231 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:09.603110    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 23:02:09.603231 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.603126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-data-volume\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:09.603231 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:09.603153    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls podName:d7a69d1d-f105-49eb-ad0e-808c10d9adae nodeName:}" failed. No retries permitted until 2026-09-10 23:02:13.603141088 +0000 UTC m=+54.294718250 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bwkkv" (UID: "d7a69d1d-f105-49eb-ad0e-808c10d9adae") : secret "cluster-monitoring-operator-tls" not found
Sep 10 23:02:09.603386 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.603311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:09.603386 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.603345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-crio-socket\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:09.704290 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.704260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:09.704452 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.704343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:09.704452 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.704371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-crio-socket\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:09.704452 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.704396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:09.704452 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:09.704417    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:02:09.704667 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.704418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:09.704667 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:09.704482    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs podName:810a3f8a-76ae-4456-82d1-2f50e10721ac nodeName:}" failed. No retries permitted until 2026-09-10 23:02:13.704459899 +0000 UTC m=+54.396037062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs") pod "router-default-8484fd4884-9th5k" (UID: "810a3f8a-76ae-4456-82d1-2f50e10721ac") : secret "router-metrics-certs-default" not found
Sep 10 23:02:09.704667 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:09.704520    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:09.704667 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.704543    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmk7\" (UniqueName: \"kubernetes.io/projected/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-kube-api-access-lgmk7\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:09.704667 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:09.704567    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle podName:810a3f8a-76ae-4456-82d1-2f50e10721ac nodeName:}" failed. No retries permitted until 2026-09-10 23:02:13.704548405 +0000 UTC m=+54.396125567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle") pod "router-default-8484fd4884-9th5k" (UID: "810a3f8a-76ae-4456-82d1-2f50e10721ac") : configmap references non-existent config key: service-ca.crt
Sep 10 23:02:09.704667 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:09.704607    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls podName:5b0e7eaf-7d3d-481d-87ae-1773ea4fc258 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:10.204595628 +0000 UTC m=+50.896172790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g8zrr" (UID: "5b0e7eaf-7d3d-481d-87ae-1773ea4fc258") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:09.704667 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.704606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-crio-socket\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:09.704912 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.704672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-data-volume\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:09.704945 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.704914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-data-volume\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:09.704982 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.704960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:09.712347 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:09.712329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmk7\" (UniqueName: \"kubernetes.io/projected/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-kube-api-access-lgmk7\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:10.210147 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:10.210056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:10.210477 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:10.210211    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:10.210477 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:10.210277    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls podName:5b0e7eaf-7d3d-481d-87ae-1773ea4fc258 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:11.210260278 +0000 UTC m=+51.901837459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g8zrr" (UID: "5b0e7eaf-7d3d-481d-87ae-1773ea4fc258") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:10.463260 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:10.463191    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5j59t_02e28b7f-e193-411b-875c-e475adab6c34/dns-node-resolver/0.log"
Sep 10 23:02:11.218883 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:11.218828    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:11.219388 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:11.218960    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:11.219388 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:11.219024    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls podName:5b0e7eaf-7d3d-481d-87ae-1773ea4fc258 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:13.219009757 +0000 UTC m=+53.910586918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g8zrr" (UID: "5b0e7eaf-7d3d-481d-87ae-1773ea4fc258") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:11.664135 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:11.664104    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n6rbk_bc7dd59a-fbd1-49b2-b886-ee2429896ddc/node-ca/0.log"
Sep 10 23:02:12.064101 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:12.064078    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hx99c_43c78e11-d941-4f95-99b2-abd00726363e/serve-healthcheck-canary/0.log"
Sep 10 23:02:13.234246 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:13.234211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:13.234681 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:13.234357    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:13.234681 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:13.234417    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls podName:5b0e7eaf-7d3d-481d-87ae-1773ea4fc258 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:17.234403286 +0000 UTC m=+57.925980452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g8zrr" (UID: "5b0e7eaf-7d3d-481d-87ae-1773ea4fc258") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:13.638064 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:13.638027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:13.638249 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:13.638202    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 23:02:13.638315 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:13.638289    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls podName:d7a69d1d-f105-49eb-ad0e-808c10d9adae nodeName:}" failed. No retries permitted until 2026-09-10 23:02:21.638265762 +0000 UTC m=+62.329842947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bwkkv" (UID: "d7a69d1d-f105-49eb-ad0e-808c10d9adae") : secret "cluster-monitoring-operator-tls" not found
Sep 10 23:02:13.739332 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:13.739300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:13.739520 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:13.739382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:13.739520 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:13.739466    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle podName:810a3f8a-76ae-4456-82d1-2f50e10721ac nodeName:}" failed. No retries permitted until 2026-09-10 23:02:21.739448797 +0000 UTC m=+62.431025959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle") pod "router-default-8484fd4884-9th5k" (UID: "810a3f8a-76ae-4456-82d1-2f50e10721ac") : configmap references non-existent config key: service-ca.crt
Sep 10 23:02:13.739520 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:13.739483    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:02:13.739645 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:13.739543    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs podName:810a3f8a-76ae-4456-82d1-2f50e10721ac nodeName:}" failed. No retries permitted until 2026-09-10 23:02:21.739528764 +0000 UTC m=+62.431105927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs") pod "router-default-8484fd4884-9th5k" (UID: "810a3f8a-76ae-4456-82d1-2f50e10721ac") : secret "router-metrics-certs-default" not found
Sep 10 23:02:17.272126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:17.272088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:17.272597 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:17.272262    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:17.272597 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:17.272332    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls podName:5b0e7eaf-7d3d-481d-87ae-1773ea4fc258 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:25.272314241 +0000 UTC m=+65.963891407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g8zrr" (UID: "5b0e7eaf-7d3d-481d-87ae-1773ea4fc258") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:20.114906 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:20.114875    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b88ks"
Sep 10 23:02:21.712124 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:21.712077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:21.712586 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:21.712219    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 23:02:21.712586 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:21.712282    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls podName:d7a69d1d-f105-49eb-ad0e-808c10d9adae nodeName:}" failed. No retries permitted until 2026-09-10 23:02:37.712267081 +0000 UTC m=+78.403844244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bwkkv" (UID: "d7a69d1d-f105-49eb-ad0e-808c10d9adae") : secret "cluster-monitoring-operator-tls" not found
Sep 10 23:02:21.813468 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:21.813437    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:21.813659 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:21.813535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:21.813659 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:21.813603    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle podName:810a3f8a-76ae-4456-82d1-2f50e10721ac nodeName:}" failed. No retries permitted until 2026-09-10 23:02:37.81358473 +0000 UTC m=+78.505161893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle") pod "router-default-8484fd4884-9th5k" (UID: "810a3f8a-76ae-4456-82d1-2f50e10721ac") : configmap references non-existent config key: service-ca.crt
Sep 10 23:02:21.813659 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:21.813628    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 23:02:21.813787 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:21.813668    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs podName:810a3f8a-76ae-4456-82d1-2f50e10721ac nodeName:}" failed. No retries permitted until 2026-09-10 23:02:37.813655194 +0000 UTC m=+78.505232356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs") pod "router-default-8484fd4884-9th5k" (UID: "810a3f8a-76ae-4456-82d1-2f50e10721ac") : secret "router-metrics-certs-default" not found
Sep 10 23:02:24.637752 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:24.637699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpth\" (UniqueName: \"kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth\") pod \"network-check-target-kr684\" (UID: \"34885b0c-c971-4056-a091-1764a294ea66\") " pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:02:24.638136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:24.637827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:02:24.638136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:24.637877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs\") pod \"network-metrics-daemon-v8sx8\" (UID: \"54f0cb94-e7d5-4b86-8602-dd8b46138dab\") " pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:02:24.641028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:24.641005    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 23:02:24.641028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:24.641017    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 23:02:24.641178 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:24.641053    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 23:02:24.648131 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:24.648105    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 23:02:24.648198 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:24.648176    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs podName:54f0cb94-e7d5-4b86-8602-dd8b46138dab nodeName:}" failed. No retries permitted until 2026-09-10 23:03:28.648157927 +0000 UTC m=+129.339735103 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs") pod "network-metrics-daemon-v8sx8" (UID: "54f0cb94-e7d5-4b86-8602-dd8b46138dab") : secret "metrics-daemon-secret" not found
Sep 10 23:02:24.650280 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:24.650260    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ab0193de-c9ef-4b5c-a865-bf880e0761d0-original-pull-secret\") pod \"global-pull-secret-syncer-xdms4\" (UID: \"ab0193de-c9ef-4b5c-a865-bf880e0761d0\") " pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:02:24.651645 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:24.651631    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 23:02:24.660172 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:24.660152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mpth\" (UniqueName: \"kubernetes.io/projected/34885b0c-c971-4056-a091-1764a294ea66-kube-api-access-5mpth\") pod \"network-check-target-kr684\" (UID: \"34885b0c-c971-4056-a091-1764a294ea66\") " pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:02:24.727776 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:24.727742    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gcwbm\""
Sep 10 23:02:24.736091 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:24.736073    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:02:24.738957 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:24.738944    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdms4"
Sep 10 23:02:25.011250 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:25.011176    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xdms4"]
Sep 10 23:02:25.013961 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:25.013938    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kr684"]
Sep 10 23:02:25.014339 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:02:25.014317    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0193de_c9ef_4b5c_a865_bf880e0761d0.slice/crio-423053a9059da01e3c147739c268c097a192dc3eb0d798e694c593683eaa8adf WatchSource:0}: Error finding container 423053a9059da01e3c147739c268c097a192dc3eb0d798e694c593683eaa8adf: Status 404 returned error can't find the container with id 423053a9059da01e3c147739c268c097a192dc3eb0d798e694c593683eaa8adf
Sep 10 23:02:25.017298 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:02:25.017276    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34885b0c_c971_4056_a091_1764a294ea66.slice/crio-90c3444789453aeb80f552b69ec4d62b28183f514d95b5e3c151767b07ea60cc WatchSource:0}: Error finding container 90c3444789453aeb80f552b69ec4d62b28183f514d95b5e3c151767b07ea60cc: Status 404 returned error can't find the container with id 90c3444789453aeb80f552b69ec4d62b28183f514d95b5e3c151767b07ea60cc
Sep 10 23:02:25.040547 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:25.040516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:02:25.040666 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:25.040651    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 23:02:25.040666 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:25.040664    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-666f85fc79-xmk8h: secret "image-registry-tls" not found
Sep 10 23:02:25.040736 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:25.040712    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls podName:d44f82f8-998f-47b3-8470-a6a8e1c66a98 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:57.040694515 +0000 UTC m=+97.732271688 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls") pod "image-registry-666f85fc79-xmk8h" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98") : secret "image-registry-tls" not found
Sep 10 23:02:25.141563 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:25.141525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:02:25.141759 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:25.141685    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 23:02:25.141827 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:25.141761    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls podName:d3aff66a-de8c-4842-902b-fe0015f1ae4b nodeName:}" failed. No retries permitted until 2026-09-10 23:02:57.1417403 +0000 UTC m=+97.833317480 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls") pod "dns-default-qq99b" (UID: "d3aff66a-de8c-4842-902b-fe0015f1ae4b") : secret "dns-default-metrics-tls" not found
Sep 10 23:02:25.200119 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:25.200080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xdms4" event={"ID":"ab0193de-c9ef-4b5c-a865-bf880e0761d0","Type":"ContainerStarted","Data":"423053a9059da01e3c147739c268c097a192dc3eb0d798e694c593683eaa8adf"}
Sep 10 23:02:25.201236 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:25.201215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kr684" event={"ID":"34885b0c-c971-4056-a091-1764a294ea66","Type":"ContainerStarted","Data":"90c3444789453aeb80f552b69ec4d62b28183f514d95b5e3c151767b07ea60cc"}
Sep 10 23:02:25.343937 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:25.343900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:25.344101 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:25.344049    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:25.344142 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:25.344115    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls podName:5b0e7eaf-7d3d-481d-87ae-1773ea4fc258 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:41.344098821 +0000 UTC m=+82.035675990 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g8zrr" (UID: "5b0e7eaf-7d3d-481d-87ae-1773ea4fc258") : secret "insights-runtime-extractor-tls" not found
Sep 10 23:02:30.217575 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:30.217539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xdms4" event={"ID":"ab0193de-c9ef-4b5c-a865-bf880e0761d0","Type":"ContainerStarted","Data":"b0dd23af8d99ddc0911ce7492e472cf18a02171067074e0c2a999780bae1d662"}
Sep 10 23:02:30.219013 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:30.218989    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kr684" event={"ID":"34885b0c-c971-4056-a091-1764a294ea66","Type":"ContainerStarted","Data":"e38498907dadb317f838493b685fcb28a1b8052d0954e8ebc496847b754bac47"}
Sep 10 23:02:30.219140 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:30.219124    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:02:30.233437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:30.233345    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xdms4" podStartSLOduration=65.602232986 podStartE2EDuration="1m10.23333354s" totalImagesPullingTime="4.631100554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:30.232947751 +0000 UTC m=+70.924524943" watchObservedRunningTime="2026-09-10 23:02:30.23333354 +0000 UTC m=+70.924910724"
Sep 10 23:02:30.248595 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:30.248562    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kr684" podStartSLOduration=65.623966022 podStartE2EDuration="1m10.248552051s" totalImagesPullingTime="4.624586029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:30.248035352 +0000 UTC m=+70.939612561" watchObservedRunningTime="2026-09-10 23:02:30.248552051 +0000 UTC m=+70.940129236"
Sep 10 23:02:37.749390 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:37.749338    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:37.751775 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:37.751756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d7a69d1d-f105-49eb-ad0e-808c10d9adae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bwkkv\" (UID: \"d7a69d1d-f105-49eb-ad0e-808c10d9adae\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:37.850153 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:37.850112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:37.850306 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:37.850291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:37.850732 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:37.850706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/810a3f8a-76ae-4456-82d1-2f50e10721ac-service-ca-bundle\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:37.852532 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:37.852514    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/810a3f8a-76ae-4456-82d1-2f50e10721ac-metrics-certs\") pod \"router-default-8484fd4884-9th5k\" (UID: \"810a3f8a-76ae-4456-82d1-2f50e10721ac\") " pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:37.953179 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:37.953148    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8mhxt\""
Sep 10 23:02:37.961956 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:37.961928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"
Sep 10 23:02:38.042987 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:38.042963    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zkt9j\""
Sep 10 23:02:38.051797 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:38.051774    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:38.229680 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:38.229616    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv"]
Sep 10 23:02:38.233193 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:02:38.233170    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a69d1d_f105_49eb_ad0e_808c10d9adae.slice/crio-1783d02a037936e7b8986674d189dd483380b28f47fc843afec93c703ee221da WatchSource:0}: Error finding container 1783d02a037936e7b8986674d189dd483380b28f47fc843afec93c703ee221da: Status 404 returned error can't find the container with id 1783d02a037936e7b8986674d189dd483380b28f47fc843afec93c703ee221da
Sep 10 23:02:38.239303 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:38.239279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv" event={"ID":"d7a69d1d-f105-49eb-ad0e-808c10d9adae","Type":"ContainerStarted","Data":"1783d02a037936e7b8986674d189dd483380b28f47fc843afec93c703ee221da"}
Sep 10 23:02:38.323711 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:38.323681    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8484fd4884-9th5k"]
Sep 10 23:02:38.326075 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:02:38.326051    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810a3f8a_76ae_4456_82d1_2f50e10721ac.slice/crio-436a549c22a845a461c1e99c100fe81200048ef9c68aac5b7f5304173dfb719e WatchSource:0}: Error finding container 436a549c22a845a461c1e99c100fe81200048ef9c68aac5b7f5304173dfb719e: Status 404 returned error can't find the container with id 436a549c22a845a461c1e99c100fe81200048ef9c68aac5b7f5304173dfb719e
Sep 10 23:02:39.248949 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:39.248912    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8484fd4884-9th5k" event={"ID":"810a3f8a-76ae-4456-82d1-2f50e10721ac","Type":"ContainerStarted","Data":"2e45c589ad466a8da42739d4e0030ed09e47f6fe77ad048b8400b614ac64fc6e"}
Sep 10 23:02:39.249424 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:39.248998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8484fd4884-9th5k" event={"ID":"810a3f8a-76ae-4456-82d1-2f50e10721ac","Type":"ContainerStarted","Data":"436a549c22a845a461c1e99c100fe81200048ef9c68aac5b7f5304173dfb719e"}
Sep 10 23:02:39.269435 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:39.269383    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-8484fd4884-9th5k" podStartSLOduration=34.269364287 podStartE2EDuration="34.269364287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:39.268576964 +0000 UTC m=+79.960154177" watchObservedRunningTime="2026-09-10 23:02:39.269364287 +0000 UTC m=+79.960941472"
Sep 10 23:02:40.052419 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:40.052388    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:40.054894 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:40.054873    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:40.257239 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:40.257142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv" event={"ID":"d7a69d1d-f105-49eb-ad0e-808c10d9adae","Type":"ContainerStarted","Data":"60664954cda22ab15a733f76d941a97fe98f7a05ac6755d581c1f17163c005cc"}
Sep 10 23:02:40.257683 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:40.257400    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:40.258795 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:40.258771    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8484fd4884-9th5k"
Sep 10 23:02:40.274533 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:40.274470    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bwkkv" podStartSLOduration=33.673937831 podStartE2EDuration="35.27445463s" totalImagesPullingTime="1.600516799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:40.273707235 +0000 UTC m=+80.965284445" watchObservedRunningTime="2026-09-10 23:02:40.27445463 +0000 UTC m=+80.966031815"
Sep 10 23:02:41.377410 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:41.377370    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:41.379671 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:41.379644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b0e7eaf-7d3d-481d-87ae-1773ea4fc258-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g8zrr\" (UID: \"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258\") " pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:41.611393 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:41.611363    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w4w8p\""
Sep 10 23:02:41.619423 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:41.619401    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g8zrr"
Sep 10 23:02:41.889835 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:41.889755    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g8zrr"]
Sep 10 23:02:41.895230 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:02:41.895206    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0e7eaf_7d3d_481d_87ae_1773ea4fc258.slice/crio-a24f0e6c85ae2d236595058938ff8ad9b90be8a556cb27101f84e7b8b03469f5 WatchSource:0}: Error finding container a24f0e6c85ae2d236595058938ff8ad9b90be8a556cb27101f84e7b8b03469f5: Status 404 returned error can't find the container with id a24f0e6c85ae2d236595058938ff8ad9b90be8a556cb27101f84e7b8b03469f5
Sep 10 23:02:42.266891 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:42.266745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g8zrr" event={"ID":"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258","Type":"ContainerStarted","Data":"a601a3c40f376fd29c7c82e63066041f03092c86e10e15eb42c5a86d703a682e"}
Sep 10 23:02:42.266891 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:42.266813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g8zrr" event={"ID":"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258","Type":"ContainerStarted","Data":"a24f0e6c85ae2d236595058938ff8ad9b90be8a556cb27101f84e7b8b03469f5"}
Sep 10 23:02:43.272582 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:43.272543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g8zrr" event={"ID":"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258","Type":"ContainerStarted","Data":"1d8bb4ffa7ec1622186938abb5499b1497cf0333716f561b2ba9d4568a72ae97"}
Sep 10 23:02:44.245853 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.245825    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"]
Sep 10 23:02:44.248745 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.248729    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"
Sep 10 23:02:44.251774 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.251725    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 23:02:44.251909 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.251773    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 23:02:44.251909 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.251741    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 23:02:44.251909 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.251810    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 23:02:44.259301 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.259276    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"]
Sep 10 23:02:44.279378 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.279342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g8zrr" event={"ID":"5b0e7eaf-7d3d-481d-87ae-1773ea4fc258","Type":"ContainerStarted","Data":"9b2b9ab1fda6372afab0ca55d3c70a61f77ab5b4f4bbc9a21b3a15c3b24074de"}
Sep 10 23:02:44.304548 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.304478    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-g8zrr" podStartSLOduration=33.165807577 podStartE2EDuration="35.304460979s" totalImagesPullingTime="2.138653402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:02:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:44.304123124 +0000 UTC m=+84.995700319" watchObservedRunningTime="2026-09-10 23:02:44.304460979 +0000 UTC m=+84.996038145"
Sep 10 23:02:44.403955 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.403871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05dc0570-e098-4268-a7df-7592c243ef97-tmp\") pod \"klusterlet-addon-workmgr-64c9696744-nnpc8\" (UID: \"05dc0570-e098-4268-a7df-7592c243ef97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"
Sep 10 23:02:44.403955 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.403914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/05dc0570-e098-4268-a7df-7592c243ef97-klusterlet-config\") pod \"klusterlet-addon-workmgr-64c9696744-nnpc8\" (UID: \"05dc0570-e098-4268-a7df-7592c243ef97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"
Sep 10 23:02:44.403955 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.403932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p2lh\" (UniqueName: \"kubernetes.io/projected/05dc0570-e098-4268-a7df-7592c243ef97-kube-api-access-2p2lh\") pod \"klusterlet-addon-workmgr-64c9696744-nnpc8\" (UID: \"05dc0570-e098-4268-a7df-7592c243ef97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"
Sep 10 23:02:44.504452 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.504424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05dc0570-e098-4268-a7df-7592c243ef97-tmp\") pod \"klusterlet-addon-workmgr-64c9696744-nnpc8\" (UID: \"05dc0570-e098-4268-a7df-7592c243ef97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"
Sep 10 23:02:44.504635 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.504468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/05dc0570-e098-4268-a7df-7592c243ef97-klusterlet-config\") pod \"klusterlet-addon-workmgr-64c9696744-nnpc8\" (UID: \"05dc0570-e098-4268-a7df-7592c243ef97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"
Sep 10 23:02:44.504635 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.504486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p2lh\" (UniqueName: \"kubernetes.io/projected/05dc0570-e098-4268-a7df-7592c243ef97-kube-api-access-2p2lh\") pod \"klusterlet-addon-workmgr-64c9696744-nnpc8\" (UID: \"05dc0570-e098-4268-a7df-7592c243ef97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"
Sep 10 23:02:44.504886 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.504869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05dc0570-e098-4268-a7df-7592c243ef97-tmp\") pod \"klusterlet-addon-workmgr-64c9696744-nnpc8\" (UID: \"05dc0570-e098-4268-a7df-7592c243ef97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"
Sep 10 23:02:44.506920 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.506893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/05dc0570-e098-4268-a7df-7592c243ef97-klusterlet-config\") pod \"klusterlet-addon-workmgr-64c9696744-nnpc8\" (UID: \"05dc0570-e098-4268-a7df-7592c243ef97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"
Sep 10 23:02:44.512986 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.512962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p2lh\" (UniqueName: \"kubernetes.io/projected/05dc0570-e098-4268-a7df-7592c243ef97-kube-api-access-2p2lh\") pod \"klusterlet-addon-workmgr-64c9696744-nnpc8\" (UID: \"05dc0570-e098-4268-a7df-7592c243ef97\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"
Sep 10 23:02:44.558641 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.558614    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"
Sep 10 23:02:44.839223 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:44.839192    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"]
Sep 10 23:02:44.842180 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:02:44.842156    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05dc0570_e098_4268_a7df_7592c243ef97.slice/crio-20400e48bca974b166aeb86ec8a15e1504ef21da52293c5b67c787ebce81738b WatchSource:0}: Error finding container 20400e48bca974b166aeb86ec8a15e1504ef21da52293c5b67c787ebce81738b: Status 404 returned error can't find the container with id 20400e48bca974b166aeb86ec8a15e1504ef21da52293c5b67c787ebce81738b
Sep 10 23:02:45.283515 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:45.283420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8" event={"ID":"05dc0570-e098-4268-a7df-7592c243ef97","Type":"ContainerStarted","Data":"20400e48bca974b166aeb86ec8a15e1504ef21da52293c5b67c787ebce81738b"}
Sep 10 23:02:49.299817 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:49.299773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8" event={"ID":"05dc0570-e098-4268-a7df-7592c243ef97","Type":"ContainerStarted","Data":"46e6f721965e4771bcd39cfc5ed4017f7bade114bbc2b94cdca019f41ef463d6"}
Sep 10 23:02:49.300201 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:49.299983    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"
Sep 10 23:02:49.301619 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:49.301602    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8"
Sep 10 23:02:49.315947 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:49.315906    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64c9696744-nnpc8" podStartSLOduration=1.54216715 podStartE2EDuration="5.315893567s" totalImagesPullingTime="3.773726417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:49.315390769 +0000 UTC m=+90.006967964" watchObservedRunningTime="2026-09-10 23:02:49.315893567 +0000 UTC m=+90.007470750"
Sep 10 23:02:51.085971 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.085935    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d467f779d-v2fr5"]
Sep 10 23:02:51.089254 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.089231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.091817 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.091790    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 23:02:51.092851 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.092833    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 23:02:51.092970 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.092861    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 23:02:51.093103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.093087    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 23:02:51.093171 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.093104    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 23:02:51.093171 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.093139    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 23:02:51.093268 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.093143    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 23:02:51.093268 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.093147    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-c5cxc\""
Sep 10 23:02:51.100029 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.100011    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d467f779d-v2fr5"]
Sep 10 23:02:51.262234 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.262184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbhtn\" (UniqueName: \"kubernetes.io/projected/078d2368-b084-4a49-a962-1f9a9a27da7b-kube-api-access-mbhtn\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.262234 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.262230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-service-ca\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.262451 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.262254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/078d2368-b084-4a49-a962-1f9a9a27da7b-console-oauth-config\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.262451 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.262316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-console-config\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.262451 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.262362    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/078d2368-b084-4a49-a962-1f9a9a27da7b-console-serving-cert\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.262451 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.262397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-oauth-serving-cert\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.363857 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.363766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/078d2368-b084-4a49-a962-1f9a9a27da7b-console-oauth-config\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.363857 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.363815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-console-config\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.363857 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.363853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/078d2368-b084-4a49-a962-1f9a9a27da7b-console-serving-cert\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.364111 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.363885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-oauth-serving-cert\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.364111 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.364034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbhtn\" (UniqueName: \"kubernetes.io/projected/078d2368-b084-4a49-a962-1f9a9a27da7b-kube-api-access-mbhtn\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.364111 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.364079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-service-ca\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.364642 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.364620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-oauth-serving-cert\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.364759 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.364684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-service-ca\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.364759 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.364699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-console-config\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.366709 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.366692    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/078d2368-b084-4a49-a962-1f9a9a27da7b-console-oauth-config\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.366791 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.366726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/078d2368-b084-4a49-a962-1f9a9a27da7b-console-serving-cert\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.373011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.372987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbhtn\" (UniqueName: \"kubernetes.io/projected/078d2368-b084-4a49-a962-1f9a9a27da7b-kube-api-access-mbhtn\") pod \"console-7d467f779d-v2fr5\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") " pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.397960 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.397940    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:02:51.668875 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:51.668802    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d467f779d-v2fr5"]
Sep 10 23:02:51.671664 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:02:51.671642    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078d2368_b084_4a49_a962_1f9a9a27da7b.slice/crio-9244b5b73a0b602d235812d83f510ec0a82770ad0bac8dac5761d55f8ef42436 WatchSource:0}: Error finding container 9244b5b73a0b602d235812d83f510ec0a82770ad0bac8dac5761d55f8ef42436: Status 404 returned error can't find the container with id 9244b5b73a0b602d235812d83f510ec0a82770ad0bac8dac5761d55f8ef42436
Sep 10 23:02:52.310766 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:52.310729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d467f779d-v2fr5" event={"ID":"078d2368-b084-4a49-a962-1f9a9a27da7b","Type":"ContainerStarted","Data":"9244b5b73a0b602d235812d83f510ec0a82770ad0bac8dac5761d55f8ef42436"}
Sep 10 23:02:55.323403 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.323370    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d467f779d-v2fr5" event={"ID":"078d2368-b084-4a49-a962-1f9a9a27da7b","Type":"ContainerStarted","Data":"cc7654c176bc1298a63344c8eb60574f9e751ec50243e7de232f880eca7e403f"}
Sep 10 23:02:55.343472 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.343417    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d467f779d-v2fr5" podStartSLOduration=1.5255277459999999 podStartE2EDuration="4.34340144s" totalImagesPullingTime="2.817873694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:55.342636734 +0000 UTC m=+96.034213938" watchObservedRunningTime="2026-09-10 23:02:55.34340144 +0000 UTC m=+96.034978685"
Sep 10 23:02:55.719818 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.719730    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"]
Sep 10 23:02:55.723063 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.723048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"
Sep 10 23:02:55.725588 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.725550    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 23:02:55.725722 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.725652    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 23:02:55.726051 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.726032    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rsfqn\""
Sep 10 23:02:55.726137 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.726058    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 23:02:55.727747 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.727721    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"]
Sep 10 23:02:55.731119 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.731098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:55.733395 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.733375    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 10 23:02:55.733593 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.733572    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"]
Sep 10 23:02:55.733822 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.733802    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 10 23:02:55.733929 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.733803    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 10 23:02:55.733999 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.733951    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wqqwc\""
Sep 10 23:02:55.742139 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.742119    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"]
Sep 10 23:02:55.752380 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.752353    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jhbpm"]
Sep 10 23:02:55.755633 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.755613    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:55.757715 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.757689    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 23:02:55.757825 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.757713    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ftsws\""
Sep 10 23:02:55.757825 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.757740    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 23:02:55.757825 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.757757    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 23:02:55.902910 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.902877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2317e5b0-b1d1-480b-baa0-944b0c684a4f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:55.902910 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.902916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3b57650-50d3-4c2e-9480-a4edf504efcd-root\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:55.903145 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.902942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc9eb081-6a81-4a89-a50f-b705af231db2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-wpx66\" (UID: \"bc9eb081-6a81-4a89-a50f-b705af231db2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"
Sep 10 23:02:55.903145 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903025    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3b57650-50d3-4c2e-9480-a4edf504efcd-metrics-client-ca\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:55.903145 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-textfile\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:55.903145 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903084    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2317e5b0-b1d1-480b-baa0-944b0c684a4f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:55.903145 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903113    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc9eb081-6a81-4a89-a50f-b705af231db2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-wpx66\" (UID: \"bc9eb081-6a81-4a89-a50f-b705af231db2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"
Sep 10 23:02:55.903372 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrlg\" (UniqueName: \"kubernetes.io/projected/bc9eb081-6a81-4a89-a50f-b705af231db2-kube-api-access-5lrlg\") pod \"openshift-state-metrics-67dcf8c979-wpx66\" (UID: \"bc9eb081-6a81-4a89-a50f-b705af231db2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"
Sep 10 23:02:55.903372 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-wtmp\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:55.903372 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:55.903372 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3b57650-50d3-4c2e-9480-a4edf504efcd-sys\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:55.903372 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-accelerators-collector-config\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:55.903372 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903344    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pntxc\" (UniqueName: \"kubernetes.io/projected/2317e5b0-b1d1-480b-baa0-944b0c684a4f-kube-api-access-pntxc\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:55.903680 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9d67\" (UniqueName: \"kubernetes.io/projected/f3b57650-50d3-4c2e-9480-a4edf504efcd-kube-api-access-z9d67\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:55.903680 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2317e5b0-b1d1-480b-baa0-944b0c684a4f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:55.903680 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2317e5b0-b1d1-480b-baa0-944b0c684a4f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:55.903680 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903528    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc9eb081-6a81-4a89-a50f-b705af231db2-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-wpx66\" (UID: \"bc9eb081-6a81-4a89-a50f-b705af231db2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"
Sep 10 23:02:55.903680 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2317e5b0-b1d1-480b-baa0-944b0c684a4f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:55.903680 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:55.903583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-tls\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.004817 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.004716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-textfile\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.004817 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.004762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2317e5b0-b1d1-480b-baa0-944b0c684a4f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:56.005049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.004862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc9eb081-6a81-4a89-a50f-b705af231db2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-wpx66\" (UID: \"bc9eb081-6a81-4a89-a50f-b705af231db2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"
Sep 10 23:02:56.005049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.004897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrlg\" (UniqueName: \"kubernetes.io/projected/bc9eb081-6a81-4a89-a50f-b705af231db2-kube-api-access-5lrlg\") pod \"openshift-state-metrics-67dcf8c979-wpx66\" (UID: \"bc9eb081-6a81-4a89-a50f-b705af231db2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"
Sep 10 23:02:56.005049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.004911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-wtmp\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.005049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.004926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.005049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.004945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3b57650-50d3-4c2e-9480-a4edf504efcd-sys\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.005049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.004977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-accelerators-collector-config\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.005049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntxc\" (UniqueName: \"kubernetes.io/projected/2317e5b0-b1d1-480b-baa0-944b0c684a4f-kube-api-access-pntxc\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:56.005049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005041    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9d67\" (UniqueName: \"kubernetes.io/projected/f3b57650-50d3-4c2e-9480-a4edf504efcd-kube-api-access-z9d67\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.005463 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005065    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2317e5b0-b1d1-480b-baa0-944b0c684a4f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:56.005463 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005091    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2317e5b0-b1d1-480b-baa0-944b0c684a4f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:56.005463 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-wtmp\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.005463 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc9eb081-6a81-4a89-a50f-b705af231db2-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-wpx66\" (UID: \"bc9eb081-6a81-4a89-a50f-b705af231db2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"
Sep 10 23:02:56.005463 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-textfile\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.005463 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005151    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3b57650-50d3-4c2e-9480-a4edf504efcd-sys\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.005806 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2317e5b0-b1d1-480b-baa0-944b0c684a4f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:56.005806 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-tls\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.005806 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2317e5b0-b1d1-480b-baa0-944b0c684a4f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:56.005806 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3b57650-50d3-4c2e-9480-a4edf504efcd-root\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.005806 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc9eb081-6a81-4a89-a50f-b705af231db2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-wpx66\" (UID: \"bc9eb081-6a81-4a89-a50f-b705af231db2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"
Sep 10 23:02:56.005806 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3b57650-50d3-4c2e-9480-a4edf504efcd-metrics-client-ca\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.006114 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2317e5b0-b1d1-480b-baa0-944b0c684a4f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:56.006114 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2317e5b0-b1d1-480b-baa0-944b0c684a4f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:56.006114 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-accelerators-collector-config\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.006114 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.005987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2317e5b0-b1d1-480b-baa0-944b0c684a4f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:56.006114 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:56.006028    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 23:02:56.006114 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.006025    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc9eb081-6a81-4a89-a50f-b705af231db2-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-wpx66\" (UID: \"bc9eb081-6a81-4a89-a50f-b705af231db2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"
Sep 10 23:02:56.006114 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.006044    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f3b57650-50d3-4c2e-9480-a4edf504efcd-root\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.006486 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:56.006185    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-tls podName:f3b57650-50d3-4c2e-9480-a4edf504efcd nodeName:}" failed. No retries permitted until 2026-09-10 23:02:56.506164841 +0000 UTC m=+97.197742018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-tls") pod "node-exporter-jhbpm" (UID: "f3b57650-50d3-4c2e-9480-a4edf504efcd") : secret "node-exporter-tls" not found
Sep 10 23:02:56.006486 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.006353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3b57650-50d3-4c2e-9480-a4edf504efcd-metrics-client-ca\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.007998 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.007969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2317e5b0-b1d1-480b-baa0-944b0c684a4f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:56.008126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.007975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.008365 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.008345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc9eb081-6a81-4a89-a50f-b705af231db2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-wpx66\" (UID: \"bc9eb081-6a81-4a89-a50f-b705af231db2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"
Sep 10 23:02:56.008424 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.008346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc9eb081-6a81-4a89-a50f-b705af231db2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-wpx66\" (UID: \"bc9eb081-6a81-4a89-a50f-b705af231db2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"
Sep 10 23:02:56.008611 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.008596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2317e5b0-b1d1-480b-baa0-944b0c684a4f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:56.013451 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.013426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrlg\" (UniqueName: \"kubernetes.io/projected/bc9eb081-6a81-4a89-a50f-b705af231db2-kube-api-access-5lrlg\") pod \"openshift-state-metrics-67dcf8c979-wpx66\" (UID: \"bc9eb081-6a81-4a89-a50f-b705af231db2\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"
Sep 10 23:02:56.013876 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.013860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9d67\" (UniqueName: \"kubernetes.io/projected/f3b57650-50d3-4c2e-9480-a4edf504efcd-kube-api-access-z9d67\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.013943 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.013878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pntxc\" (UniqueName: \"kubernetes.io/projected/2317e5b0-b1d1-480b-baa0-944b0c684a4f-kube-api-access-pntxc\") pod \"kube-state-metrics-59fbdd77d-9jcw2\" (UID: \"2317e5b0-b1d1-480b-baa0-944b0c684a4f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:56.031826 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.031792    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"
Sep 10 23:02:56.039920 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.039897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"
Sep 10 23:02:56.313324 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.313292    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66"]
Sep 10 23:02:56.316334 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:02:56.316311    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9eb081_6a81_4a89_a50f_b705af231db2.slice/crio-9e55a28768979e3b2ba3c179b6b3b8b948f3cd3f5adb1015a0ebce5ed09828e6 WatchSource:0}: Error finding container 9e55a28768979e3b2ba3c179b6b3b8b948f3cd3f5adb1015a0ebce5ed09828e6: Status 404 returned error can't find the container with id 9e55a28768979e3b2ba3c179b6b3b8b948f3cd3f5adb1015a0ebce5ed09828e6
Sep 10 23:02:56.319286 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.319269    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2"]
Sep 10 23:02:56.321746 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:02:56.321726    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2317e5b0_b1d1_480b_baa0_944b0c684a4f.slice/crio-044c4678a3bef8fc0c7fc8f3f8c1ce156ad1282c21f37881ed521a5a0ffb2107 WatchSource:0}: Error finding container 044c4678a3bef8fc0c7fc8f3f8c1ce156ad1282c21f37881ed521a5a0ffb2107: Status 404 returned error can't find the container with id 044c4678a3bef8fc0c7fc8f3f8c1ce156ad1282c21f37881ed521a5a0ffb2107
Sep 10 23:02:56.327324 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.327302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2" event={"ID":"2317e5b0-b1d1-480b-baa0-944b0c684a4f","Type":"ContainerStarted","Data":"044c4678a3bef8fc0c7fc8f3f8c1ce156ad1282c21f37881ed521a5a0ffb2107"}
Sep 10 23:02:56.328530 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.328512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66" event={"ID":"bc9eb081-6a81-4a89-a50f-b705af231db2","Type":"ContainerStarted","Data":"9e55a28768979e3b2ba3c179b6b3b8b948f3cd3f5adb1015a0ebce5ed09828e6"}
Sep 10 23:02:56.510524 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.510475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-tls\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:56.510798 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:56.510777    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 23:02:56.510870 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:56.510835    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-tls podName:f3b57650-50d3-4c2e-9480-a4edf504efcd nodeName:}" failed. No retries permitted until 2026-09-10 23:02:57.510821523 +0000 UTC m=+98.202398686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-tls") pod "node-exporter-jhbpm" (UID: "f3b57650-50d3-4c2e-9480-a4edf504efcd") : secret "node-exporter-tls" not found
Sep 10 23:02:56.802571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.802479    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 23:02:56.806391 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.806369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:56.809252 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.809225    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 23:02:56.809252 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.809237    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 23:02:56.809419 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.809264    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 23:02:56.809419 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.809225    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 23:02:56.809554 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.809450    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nrrt5\""
Sep 10 23:02:56.809976 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.809609    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 23:02:56.809976 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.809834    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 23:02:56.809976 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.809903    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 23:02:56.810173 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.810041    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 23:02:56.810173 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.810090    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 23:02:56.821380 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.821358    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 23:02:56.913825 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.913789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:56.914014 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.913840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:56.914014 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.913973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-web-config\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:56.914014 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.914005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:56.914180 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.914061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:56.914180 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.914089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:56.914180 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.914156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/984b8a72-f663-4249-8fe7-c5df654f8d42-tls-assets\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:56.914332 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.914190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/984b8a72-f663-4249-8fe7-c5df654f8d42-config-out\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:56.914332 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.914216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:56.914332 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.914283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:56.914616 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.914592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vf6x\" (UniqueName: \"kubernetes.io/projected/984b8a72-f663-4249-8fe7-c5df654f8d42-kube-api-access-6vf6x\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:56.914697 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.914638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-config-volume\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:56.914697 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:56.914664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.016025 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.015992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-web-config\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.016025 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.016036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.016257 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.016081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.016257 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.016106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.016257 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.016149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/984b8a72-f663-4249-8fe7-c5df654f8d42-tls-assets\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.016257 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.016176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/984b8a72-f663-4249-8fe7-c5df654f8d42-config-out\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.016257 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.016203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.016573 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.016263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.016573 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.016297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vf6x\" (UniqueName: \"kubernetes.io/projected/984b8a72-f663-4249-8fe7-c5df654f8d42-kube-api-access-6vf6x\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.016573 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.016330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-config-volume\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.016573 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.016356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.016573 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.016395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.016573 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.016429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.017755 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:02:57.017730    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-trusted-ca-bundle podName:984b8a72-f663-4249-8fe7-c5df654f8d42 nodeName:}" failed. No retries permitted until 2026-09-10 23:02:57.517705907 +0000 UTC m=+98.209283082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "984b8a72-f663-4249-8fe7-c5df654f8d42") : configmap references non-existent config key: ca-bundle.crt
Sep 10 23:02:57.018051 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.018029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.019038 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.019015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.019862 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.019818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.023721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.023680    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/984b8a72-f663-4249-8fe7-c5df654f8d42-config-out\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.025049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.025031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.025162 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.025135    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.025247 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.025226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/984b8a72-f663-4249-8fe7-c5df654f8d42-tls-assets\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.025380 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.025355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.025380 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.025374    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.025681 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.025662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-config-volume\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.026759 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.026719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vf6x\" (UniqueName: \"kubernetes.io/projected/984b8a72-f663-4249-8fe7-c5df654f8d42-kube-api-access-6vf6x\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.027356 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.027338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-web-config\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.117623 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.117477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:02:57.120129 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.120102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls\") pod \"image-registry-666f85fc79-xmk8h\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") " pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:02:57.218831 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.218792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:02:57.221084 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.221057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3aff66a-de8c-4842-902b-fe0015f1ae4b-metrics-tls\") pod \"dns-default-qq99b\" (UID: \"d3aff66a-de8c-4842-902b-fe0015f1ae4b\") " pod="openshift-dns/dns-default-qq99b"
Sep 10 23:02:57.334880 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.334842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66" event={"ID":"bc9eb081-6a81-4a89-a50f-b705af231db2","Type":"ContainerStarted","Data":"6fbfd6f1370d1293003a49b0184241eafa593eace5eb9b650a01e4ad7990a58d"}
Sep 10 23:02:57.335280 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.334937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66" event={"ID":"bc9eb081-6a81-4a89-a50f-b705af231db2","Type":"ContainerStarted","Data":"4a011bf879ae236ec0a54e5cd824a9718d004dc7815b9f52e4d4cd211399ea79"}
Sep 10 23:02:57.414367 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.414297    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-78nrb\""
Sep 10 23:02:57.422224 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.422198    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:02:57.423066 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.423042    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-497ss\""
Sep 10 23:02:57.432155 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.432136    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qq99b"
Sep 10 23:02:57.520848 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.520815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-tls\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:57.521010 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.520859    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.521790 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.521761    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.523115 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.523091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f3b57650-50d3-4c2e-9480-a4edf504efcd-node-exporter-tls\") pod \"node-exporter-jhbpm\" (UID: \"f3b57650-50d3-4c2e-9480-a4edf504efcd\") " pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:57.564134 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.564093    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jhbpm"
Sep 10 23:02:57.719397 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:57.719329    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:02:57.757757 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:02:57.757724    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b57650_50d3_4c2e_9480_a4edf504efcd.slice/crio-f6415df72a7af59e5c470a1a7cffde5236c12a67f3790cdd871b6d6365e81020 WatchSource:0}: Error finding container f6415df72a7af59e5c470a1a7cffde5236c12a67f3790cdd871b6d6365e81020: Status 404 returned error can't find the container with id f6415df72a7af59e5c470a1a7cffde5236c12a67f3790cdd871b6d6365e81020
Sep 10 23:02:58.022230 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.022155    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qq99b"]
Sep 10 23:02:58.026367 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:02:58.026326    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3aff66a_de8c_4842_902b_fe0015f1ae4b.slice/crio-5467332782fc6c6fdefa269db8e155cf1a91d28cd8589781afd4a67936a7ef1a WatchSource:0}: Error finding container 5467332782fc6c6fdefa269db8e155cf1a91d28cd8589781afd4a67936a7ef1a: Status 404 returned error can't find the container with id 5467332782fc6c6fdefa269db8e155cf1a91d28cd8589781afd4a67936a7ef1a
Sep 10 23:02:58.035592 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.035571    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-666f85fc79-xmk8h"]
Sep 10 23:02:58.038796 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:02:58.038771    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44f82f8_998f_47b3_8470_a6a8e1c66a98.slice/crio-8a67a666f39182f9b2791a844375fb40a0b8cdfee20c9997db772f072a8c9f04 WatchSource:0}: Error finding container 8a67a666f39182f9b2791a844375fb40a0b8cdfee20c9997db772f072a8c9f04: Status 404 returned error can't find the container with id 8a67a666f39182f9b2791a844375fb40a0b8cdfee20c9997db772f072a8c9f04
Sep 10 23:02:58.052968 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.052940    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 23:02:58.054864 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:02:58.054839    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984b8a72_f663_4249_8fe7_c5df654f8d42.slice/crio-134339db5c5d58ce9529aea84a25b98222ef7dc76a637bee26e4ba99f1c2d96b WatchSource:0}: Error finding container 134339db5c5d58ce9529aea84a25b98222ef7dc76a637bee26e4ba99f1c2d96b: Status 404 returned error can't find the container with id 134339db5c5d58ce9529aea84a25b98222ef7dc76a637bee26e4ba99f1c2d96b
Sep 10 23:02:58.341012 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.340972    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerStarted","Data":"134339db5c5d58ce9529aea84a25b98222ef7dc76a637bee26e4ba99f1c2d96b"}
Sep 10 23:02:58.342551 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.342522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jhbpm" event={"ID":"f3b57650-50d3-4c2e-9480-a4edf504efcd","Type":"ContainerStarted","Data":"f6415df72a7af59e5c470a1a7cffde5236c12a67f3790cdd871b6d6365e81020"}
Sep 10 23:02:58.343941 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.343918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qq99b" event={"ID":"d3aff66a-de8c-4842-902b-fe0015f1ae4b","Type":"ContainerStarted","Data":"5467332782fc6c6fdefa269db8e155cf1a91d28cd8589781afd4a67936a7ef1a"}
Sep 10 23:02:58.346862 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.346672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2" event={"ID":"2317e5b0-b1d1-480b-baa0-944b0c684a4f","Type":"ContainerStarted","Data":"a6c5c7809102266cae9e343d496010d82068e352366831e61d933d4cad159de0"}
Sep 10 23:02:58.346862 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.346726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2" event={"ID":"2317e5b0-b1d1-480b-baa0-944b0c684a4f","Type":"ContainerStarted","Data":"73f94434fdea114b827c2b7be648642598c7e707e104cba776b8434b25cc2e62"}
Sep 10 23:02:58.346862 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.346755    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2" event={"ID":"2317e5b0-b1d1-480b-baa0-944b0c684a4f","Type":"ContainerStarted","Data":"eb2087e5eafde6343ff7add65689785b2b05d2fbd336a32f5b8a9d300e69dc12"}
Sep 10 23:02:58.349544 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.349447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66" event={"ID":"bc9eb081-6a81-4a89-a50f-b705af231db2","Type":"ContainerStarted","Data":"347b5810e46c1392786bf2d2fdf9c02d6cdfa43b7b250fbe3a53e74025761ed8"}
Sep 10 23:02:58.351188 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.351166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666f85fc79-xmk8h" event={"ID":"d44f82f8-998f-47b3-8470-a6a8e1c66a98","Type":"ContainerStarted","Data":"05fc27a7c37428220c3b7b81020a47cc545b048737d1834a2902ac21987e3f89"}
Sep 10 23:02:58.351271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.351207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666f85fc79-xmk8h" event={"ID":"d44f82f8-998f-47b3-8470-a6a8e1c66a98","Type":"ContainerStarted","Data":"8a67a666f39182f9b2791a844375fb40a0b8cdfee20c9997db772f072a8c9f04"}
Sep 10 23:02:58.351420 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.351379    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:02:58.365756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.365710    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9jcw2" podStartSLOduration=1.932757519 podStartE2EDuration="3.36569249s" totalImagesPullingTime="1.432934971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:58.364635958 +0000 UTC m=+99.056213169" watchObservedRunningTime="2026-09-10 23:02:58.36569249 +0000 UTC m=+99.057269674"
Sep 10 23:02:58.385136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.385081    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-666f85fc79-xmk8h" podStartSLOduration=98.385059975 podStartE2EDuration="1m38.385059975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:58.38451645 +0000 UTC m=+99.076093651" watchObservedRunningTime="2026-09-10 23:02:58.385059975 +0000 UTC m=+99.076637163"
Sep 10 23:02:58.401478 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:58.401421    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-wpx66" podStartSLOduration=2.073119801 podStartE2EDuration="3.40140177s" totalImagesPullingTime="1.328281969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:58.400768127 +0000 UTC m=+99.092345337" watchObservedRunningTime="2026-09-10 23:02:58.40140177 +0000 UTC m=+99.092978954"
Sep 10 23:02:59.357425 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:59.357393    2563 generic.go:356] "Generic (PLEG): container finished" podID="f3b57650-50d3-4c2e-9480-a4edf504efcd" containerID="22ba7dbde4ee476e5a552a9cab6a2fda49f26e11e98a1ca1776bcfe64916d45e" exitCode=0
Sep 10 23:02:59.357904 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:02:59.357485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jhbpm" event={"ID":"f3b57650-50d3-4c2e-9480-a4edf504efcd","Type":"ContainerDied","Data":"22ba7dbde4ee476e5a552a9cab6a2fda49f26e11e98a1ca1776bcfe64916d45e"}
Sep 10 23:03:00.362961 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.362926    2563 generic.go:356] "Generic (PLEG): container finished" podID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerID="71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642" exitCode=0
Sep 10 23:03:00.363375 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.363013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerDied","Data":"71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642"}
Sep 10 23:03:00.365372 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.365345    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jhbpm" event={"ID":"f3b57650-50d3-4c2e-9480-a4edf504efcd","Type":"ContainerStarted","Data":"ce735d18256bdbe2acad0074e1dba4daa797dd66cfbdfae94d98a323114a79db"}
Sep 10 23:03:00.365475 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.365383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jhbpm" event={"ID":"f3b57650-50d3-4c2e-9480-a4edf504efcd","Type":"ContainerStarted","Data":"408374e249cfff95277343ba8a3cf65eb2d1a6caa1eae7908618841323a11f13"}
Sep 10 23:03:00.367230 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.367206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qq99b" event={"ID":"d3aff66a-de8c-4842-902b-fe0015f1ae4b","Type":"ContainerStarted","Data":"5a52ef67de762b5617f2ca714833c2bec0e272aeb6ed90cb969b154308b60b49"}
Sep 10 23:03:00.367317 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.367253    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qq99b" event={"ID":"d3aff66a-de8c-4842-902b-fe0015f1ae4b","Type":"ContainerStarted","Data":"b501f4c0aedd42191b1fddd318a434edb3af789fa7130dbb7c16865eef2da3b7"}
Sep 10 23:03:00.367317 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.367296    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qq99b"
Sep 10 23:03:00.412336 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.412285    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qq99b" podStartSLOduration=65.487502001 podStartE2EDuration="1m7.412268793s" totalImagesPullingTime="1.924766792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:00.410451008 +0000 UTC m=+101.102028212" watchObservedRunningTime="2026-09-10 23:03:00.412268793 +0000 UTC m=+101.103845977"
Sep 10 23:03:00.433547 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.433472    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jhbpm" podStartSLOduration=4.417813671 podStartE2EDuration="5.433454268s" totalImagesPullingTime="951.661916ms" totalInitContainerRuntime="63.978681ms" isStatefulPod=true podCreationTimestamp="2026-09-10 23:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:00.432960015 +0000 UTC m=+101.124537229" watchObservedRunningTime="2026-09-10 23:03:00.433454268 +0000 UTC m=+101.125031453"
Sep 10 23:03:00.513479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.513433    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w"]
Sep 10 23:03:00.518172 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.518150    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w"
Sep 10 23:03:00.520610 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.520587    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 10 23:03:00.521000 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.520677    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xgzsq\""
Sep 10 23:03:00.522073 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.522049    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d467f779d-v2fr5"]
Sep 10 23:03:00.525249 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.525226    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w"]
Sep 10 23:03:00.551200 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.551173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5ba300f-7259-4027-b26d-4eab1b9bea4d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-qjn5w\" (UID: \"d5ba300f-7259-4027-b26d-4eab1b9bea4d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w"
Sep 10 23:03:00.551365 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.551225    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt4lh\" (UniqueName: \"kubernetes.io/projected/d5ba300f-7259-4027-b26d-4eab1b9bea4d-kube-api-access-gt4lh\") pod \"monitoring-plugin-5d85778b9d-qjn5w\" (UID: \"d5ba300f-7259-4027-b26d-4eab1b9bea4d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w"
Sep 10 23:03:00.652346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.652269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5ba300f-7259-4027-b26d-4eab1b9bea4d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-qjn5w\" (UID: \"d5ba300f-7259-4027-b26d-4eab1b9bea4d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w"
Sep 10 23:03:00.652346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.652318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt4lh\" (UniqueName: \"kubernetes.io/projected/d5ba300f-7259-4027-b26d-4eab1b9bea4d-kube-api-access-gt4lh\") pod \"monitoring-plugin-5d85778b9d-qjn5w\" (UID: \"d5ba300f-7259-4027-b26d-4eab1b9bea4d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w"
Sep 10 23:03:00.652574 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:03:00.652424    2563 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 10 23:03:00.652574 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:03:00.652521    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5ba300f-7259-4027-b26d-4eab1b9bea4d-monitoring-plugin-cert podName:d5ba300f-7259-4027-b26d-4eab1b9bea4d nodeName:}" failed. No retries permitted until 2026-09-10 23:03:01.152481696 +0000 UTC m=+101.844058862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d5ba300f-7259-4027-b26d-4eab1b9bea4d-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-qjn5w" (UID: "d5ba300f-7259-4027-b26d-4eab1b9bea4d") : secret "monitoring-plugin-cert" not found
Sep 10 23:03:00.660628 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.660606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt4lh\" (UniqueName: \"kubernetes.io/projected/d5ba300f-7259-4027-b26d-4eab1b9bea4d-kube-api-access-gt4lh\") pod \"monitoring-plugin-5d85778b9d-qjn5w\" (UID: \"d5ba300f-7259-4027-b26d-4eab1b9bea4d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w"
Sep 10 23:03:00.928836 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.928741    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-578d599455-5jl7t"]
Sep 10 23:03:00.932702 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.932675    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:00.936110 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.936083    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-flbgh\""
Sep 10 23:03:00.937185 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.936980    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 10 23:03:00.937185 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.937063    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 10 23:03:00.937986 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.937389    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 10 23:03:00.937986 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.937541    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 10 23:03:00.947718 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.947515    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 10 23:03:00.952350 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.952325    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-578d599455-5jl7t"]
Sep 10 23:03:00.954901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.954761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7554854d-c19c-4597-9046-6fc8c32ec82f-federate-client-tls\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:00.954901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.954801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7554854d-c19c-4597-9046-6fc8c32ec82f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:00.954901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.954843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7554854d-c19c-4597-9046-6fc8c32ec82f-serving-certs-ca-bundle\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:00.955121 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.954928    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7554854d-c19c-4597-9046-6fc8c32ec82f-secret-telemeter-client\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:00.955121 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.955019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7554854d-c19c-4597-9046-6fc8c32ec82f-metrics-client-ca\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:00.955121 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.954347    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 10 23:03:00.955121 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.955060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7554854d-c19c-4597-9046-6fc8c32ec82f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:00.955121 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.955088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766k6\" (UniqueName: \"kubernetes.io/projected/7554854d-c19c-4597-9046-6fc8c32ec82f-kube-api-access-766k6\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:00.955368 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:00.955122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7554854d-c19c-4597-9046-6fc8c32ec82f-telemeter-client-tls\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.055806 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.055767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7554854d-c19c-4597-9046-6fc8c32ec82f-federate-client-tls\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.055997 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.055818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7554854d-c19c-4597-9046-6fc8c32ec82f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.055997 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.055855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7554854d-c19c-4597-9046-6fc8c32ec82f-serving-certs-ca-bundle\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.055997 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.055910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7554854d-c19c-4597-9046-6fc8c32ec82f-secret-telemeter-client\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.055997 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.055950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7554854d-c19c-4597-9046-6fc8c32ec82f-metrics-client-ca\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.055997 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.055979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7554854d-c19c-4597-9046-6fc8c32ec82f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.056238 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.056009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766k6\" (UniqueName: \"kubernetes.io/projected/7554854d-c19c-4597-9046-6fc8c32ec82f-kube-api-access-766k6\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.056238 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.056074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7554854d-c19c-4597-9046-6fc8c32ec82f-telemeter-client-tls\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.056972 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.056945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7554854d-c19c-4597-9046-6fc8c32ec82f-metrics-client-ca\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.056972 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.056968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7554854d-c19c-4597-9046-6fc8c32ec82f-serving-certs-ca-bundle\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.057236 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.057024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7554854d-c19c-4597-9046-6fc8c32ec82f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.059094 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.059069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7554854d-c19c-4597-9046-6fc8c32ec82f-federate-client-tls\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.059186 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.059080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7554854d-c19c-4597-9046-6fc8c32ec82f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.059186 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.059139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7554854d-c19c-4597-9046-6fc8c32ec82f-secret-telemeter-client\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.060325 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.060298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7554854d-c19c-4597-9046-6fc8c32ec82f-telemeter-client-tls\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.064038 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.064014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-766k6\" (UniqueName: \"kubernetes.io/projected/7554854d-c19c-4597-9046-6fc8c32ec82f-kube-api-access-766k6\") pod \"telemeter-client-578d599455-5jl7t\" (UID: \"7554854d-c19c-4597-9046-6fc8c32ec82f\") " pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.157333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.157300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5ba300f-7259-4027-b26d-4eab1b9bea4d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-qjn5w\" (UID: \"d5ba300f-7259-4027-b26d-4eab1b9bea4d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w"
Sep 10 23:03:01.159677 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.159655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5ba300f-7259-4027-b26d-4eab1b9bea4d-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-qjn5w\" (UID: \"d5ba300f-7259-4027-b26d-4eab1b9bea4d\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w"
Sep 10 23:03:01.225047 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.224967    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kr684"
Sep 10 23:03:01.247887 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.247855    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-578d599455-5jl7t"
Sep 10 23:03:01.398650 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.398603    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:03:01.427820 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.427784    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w"
Sep 10 23:03:01.542145 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.542110    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-578d599455-5jl7t"]
Sep 10 23:03:01.699860 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.699832    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w"]
Sep 10 23:03:01.929011 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:03:01.928976    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7554854d_c19c_4597_9046_6fc8c32ec82f.slice/crio-a42e85fb4225558a9e8611e4d634e5385c48d4354cf7a33d72ceaa516080ba3c WatchSource:0}: Error finding container a42e85fb4225558a9e8611e4d634e5385c48d4354cf7a33d72ceaa516080ba3c: Status 404 returned error can't find the container with id a42e85fb4225558a9e8611e4d634e5385c48d4354cf7a33d72ceaa516080ba3c
Sep 10 23:03:01.929437 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:03:01.929421    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ba300f_7259_4027_b26d_4eab1b9bea4d.slice/crio-78a975806c3fe55c9562fcd1d8ca1eea414c7191b7a438e5ed7e3f9d15c81210 WatchSource:0}: Error finding container 78a975806c3fe55c9562fcd1d8ca1eea414c7191b7a438e5ed7e3f9d15c81210: Status 404 returned error can't find the container with id 78a975806c3fe55c9562fcd1d8ca1eea414c7191b7a438e5ed7e3f9d15c81210
Sep 10 23:03:01.986419 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:01.986380    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:03:02.009272 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.009238    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:03:02.009438 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.009404    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.011658 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.011629    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 23:03:02.011767 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.011737    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-12ks42kfv2ac3\""
Sep 10 23:03:02.011970 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.011955    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 23:03:02.012302 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.012229    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7z8pw\""
Sep 10 23:03:02.012302 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.012269    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 23:03:02.012302 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.012267    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 23:03:02.012532 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.012391    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 23:03:02.012736 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.012718    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 23:03:02.012736 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.012729    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 23:03:02.012870 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.012747    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 23:03:02.012870 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.012730    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 23:03:02.012999 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.012982    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 23:03:02.013209 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.013190    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 23:03:02.013209 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.013206    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 23:03:02.013487 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.013440    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 23:03:02.013624 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.013545    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 23:03:02.019735 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.019561    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 23:03:02.066419 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.066576 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066422    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e4cd43c-288a-44a1-a70e-3e4be611b868-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.066576 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.066576 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.066576 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.066576 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e4cd43c-288a-44a1-a70e-3e4be611b868-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.066851 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.066851 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.066851 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066709    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.066851 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.066851 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.066851 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.066851 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066815    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdt6\" (UniqueName: \"kubernetes.io/projected/6e4cd43c-288a-44a1-a70e-3e4be611b868-kube-api-access-ctdt6\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.066851 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.067177 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.067177 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.067177 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.067177 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.066993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.067177 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.067022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.067177 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.067052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-config\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.168461 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.168627 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.168627 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-config\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.168627 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.168771 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e4cd43c-288a-44a1-a70e-3e4be611b868-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.168771 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.168771 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.168771 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.168771 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e4cd43c-288a-44a1-a70e-3e4be611b868-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.169003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.169003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.169003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.169003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.169003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.169003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.169003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.168984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdt6\" (UniqueName: \"kubernetes.io/projected/6e4cd43c-288a-44a1-a70e-3e4be611b868-kube-api-access-ctdt6\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.169756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.169016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.169756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.169041    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.169756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.169084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.169756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.169105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.169756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.169122    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.169756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.169568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.169756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.169720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.171346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.171022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.172937 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.172008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-web-config\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.172937 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.172418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.172937 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.172732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.172937 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.172898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.173303 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.173279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.173403 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.173302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.173567 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.173546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e4cd43c-288a-44a1-a70e-3e4be611b868-config-out\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.174107 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.173635    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.174107 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.173963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-config\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.174107 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.174066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e4cd43c-288a-44a1-a70e-3e4be611b868-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.174299 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.174144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.174376 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.174351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.174670 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.174648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.175152 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.175131    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.175537 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.175485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.181072 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.181029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdt6\" (UniqueName: \"kubernetes.io/projected/6e4cd43c-288a-44a1-a70e-3e4be611b868-kube-api-access-ctdt6\") pod \"prometheus-k8s-0\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.324540 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.324516    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:02.378323 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.378289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w" event={"ID":"d5ba300f-7259-4027-b26d-4eab1b9bea4d","Type":"ContainerStarted","Data":"78a975806c3fe55c9562fcd1d8ca1eea414c7191b7a438e5ed7e3f9d15c81210"}
Sep 10 23:03:02.379612 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.379577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-578d599455-5jl7t" event={"ID":"7554854d-c19c-4597-9046-6fc8c32ec82f","Type":"ContainerStarted","Data":"a42e85fb4225558a9e8611e4d634e5385c48d4354cf7a33d72ceaa516080ba3c"}
Sep 10 23:03:02.619778 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:02.619752    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:03:02.625582 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:03:02.625551    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4cd43c_288a_44a1_a70e_3e4be611b868.slice/crio-b90d76223303a14f05617388eaae60eac9b70869e6e56a83177fff9fd3e0ab78 WatchSource:0}: Error finding container b90d76223303a14f05617388eaae60eac9b70869e6e56a83177fff9fd3e0ab78: Status 404 returned error can't find the container with id b90d76223303a14f05617388eaae60eac9b70869e6e56a83177fff9fd3e0ab78
Sep 10 23:03:03.389484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:03.389233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerStarted","Data":"08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0"}
Sep 10 23:03:03.389484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:03.389326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerStarted","Data":"c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b"}
Sep 10 23:03:03.389484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:03.389366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerStarted","Data":"41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee"}
Sep 10 23:03:03.389484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:03.389404    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerStarted","Data":"b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9"}
Sep 10 23:03:03.389484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:03.389438    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerStarted","Data":"3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609"}
Sep 10 23:03:03.391604 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:03.391578    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerID="6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd" exitCode=0
Sep 10 23:03:03.391696 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:03.391636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerDied","Data":"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"}
Sep 10 23:03:03.391696 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:03.391684    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerStarted","Data":"b90d76223303a14f05617388eaae60eac9b70869e6e56a83177fff9fd3e0ab78"}
Sep 10 23:03:05.402950 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:05.402896    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-578d599455-5jl7t" event={"ID":"7554854d-c19c-4597-9046-6fc8c32ec82f","Type":"ContainerStarted","Data":"07ec85904942c582ba0a6fbc189d3c843fa62436619d62a2e20c37972ddb587b"}
Sep 10 23:03:05.403439 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:05.402978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-578d599455-5jl7t" event={"ID":"7554854d-c19c-4597-9046-6fc8c32ec82f","Type":"ContainerStarted","Data":"9bad2099af22429e2c29ec1e34153a1132c7b3371ae807b3dc88aa114a4a7d7c"}
Sep 10 23:03:05.403439 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:05.403010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-578d599455-5jl7t" event={"ID":"7554854d-c19c-4597-9046-6fc8c32ec82f","Type":"ContainerStarted","Data":"f9611a7680eb0edb8fbcc984e308e6b7e098c71e5ab9ff28aa6700f0e3edafb4"}
Sep 10 23:03:05.406884 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:05.406825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerStarted","Data":"39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a"}
Sep 10 23:03:05.408595 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:05.408572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w" event={"ID":"d5ba300f-7259-4027-b26d-4eab1b9bea4d","Type":"ContainerStarted","Data":"209995ab8476f7309803afa0d0e2cf4b07cee34528de9f01d812a9128ecf5571"}
Sep 10 23:03:05.408814 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:05.408789    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w"
Sep 10 23:03:05.413855 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:05.413832    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w"
Sep 10 23:03:05.424096 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:05.424052    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-578d599455-5jl7t" podStartSLOduration=2.8761060130000002 podStartE2EDuration="5.424040184s" totalImagesPullingTime="2.547934171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:05.423118554 +0000 UTC m=+106.114695773" watchObservedRunningTime="2026-09-10 23:03:05.424040184 +0000 UTC m=+106.115617367"
Sep 10 23:03:05.440441 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:05.440384    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-qjn5w" podStartSLOduration=2.89810111 podStartE2EDuration="5.440364507s" totalImagesPullingTime="2.542263397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:03:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:05.439519314 +0000 UTC m=+106.131096512" watchObservedRunningTime="2026-09-10 23:03:05.440364507 +0000 UTC m=+106.131941695"
Sep 10 23:03:05.469924 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:05.469862    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.5627896420000003 podStartE2EDuration="9.469838455s" totalImagesPullingTime="5.884472991s" totalInitContainerRuntime="22.575822ms" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:05.468872462 +0000 UTC m=+106.160449672" watchObservedRunningTime="2026-09-10 23:03:05.469838455 +0000 UTC m=+106.161415640"
Sep 10 23:03:06.201064 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:06.201024    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-666f85fc79-xmk8h"]
Sep 10 23:03:07.421558 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:07.421524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerStarted","Data":"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"}
Sep 10 23:03:07.421915 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:07.421624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerStarted","Data":"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"}
Sep 10 23:03:09.434891 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:09.434852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerStarted","Data":"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"}
Sep 10 23:03:09.435268 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:09.434915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerStarted","Data":"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"}
Sep 10 23:03:09.435268 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:09.434946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerStarted","Data":"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"}
Sep 10 23:03:09.435268 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:09.434974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerStarted","Data":"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"}
Sep 10 23:03:09.462881 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:09.462819    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.958104515 podStartE2EDuration="8.462799222s" totalImagesPullingTime="5.487965919s" totalInitContainerRuntime="16.728788ms" isStatefulPod=false podCreationTimestamp="2026-09-10 23:03:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:09.460395465 +0000 UTC m=+110.151972648" watchObservedRunningTime="2026-09-10 23:03:09.462799222 +0000 UTC m=+110.154376406"
Sep 10 23:03:10.373719 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:10.373690    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qq99b"
Sep 10 23:03:12.324836 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:12.324796    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:03:16.207397 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:16.207368    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:03:19.475946 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:19.475910    2563 generic.go:356] "Generic (PLEG): container finished" podID="77ac1f77-479a-4611-b90c-c5ffb2242a9f" containerID="943bb2d652844a5acfa835e658b48dfb9ac880e910641c42e0fa06f98bb73cd8" exitCode=0
Sep 10 23:03:19.476326 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:19.475982    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-96fls" event={"ID":"77ac1f77-479a-4611-b90c-c5ffb2242a9f","Type":"ContainerDied","Data":"943bb2d652844a5acfa835e658b48dfb9ac880e910641c42e0fa06f98bb73cd8"}
Sep 10 23:03:19.476367 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:19.476335    2563 scope.go:117] "RemoveContainer" containerID="943bb2d652844a5acfa835e658b48dfb9ac880e910641c42e0fa06f98bb73cd8"
Sep 10 23:03:20.113996 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:20.113967    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8484fd4884-9th5k_810a3f8a-76ae-4456-82d1-2f50e10721ac/router/0.log"
Sep 10 23:03:20.119840 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:20.119799    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hx99c_43c78e11-d941-4f95-99b2-abd00726363e/serve-healthcheck-canary/0.log"
Sep 10 23:03:20.481430 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:20.481339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-96fls" event={"ID":"77ac1f77-479a-4611-b90c-c5ffb2242a9f","Type":"ContainerStarted","Data":"b61d76dfae5ff01bc952de26a693929bcb982aba858512305e3f6dad17f837ec"}
Sep 10 23:03:25.541945 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.541907    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d467f779d-v2fr5" podUID="078d2368-b084-4a49-a962-1f9a9a27da7b" containerName="console" containerID="cri-o://cc7654c176bc1298a63344c8eb60574f9e751ec50243e7de232f880eca7e403f" gracePeriod=15
Sep 10 23:03:25.774860 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.774834    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:03:25.892801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.892757    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-console-config\") pod \"078d2368-b084-4a49-a962-1f9a9a27da7b\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") "
Sep 10 23:03:25.893004 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.892863    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-oauth-serving-cert\") pod \"078d2368-b084-4a49-a962-1f9a9a27da7b\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") "
Sep 10 23:03:25.893004 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.892895    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/078d2368-b084-4a49-a962-1f9a9a27da7b-console-serving-cert\") pod \"078d2368-b084-4a49-a962-1f9a9a27da7b\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") "
Sep 10 23:03:25.893004 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.892922    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/078d2368-b084-4a49-a962-1f9a9a27da7b-console-oauth-config\") pod \"078d2368-b084-4a49-a962-1f9a9a27da7b\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") "
Sep 10 23:03:25.893004 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.892961    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbhtn\" (UniqueName: \"kubernetes.io/projected/078d2368-b084-4a49-a962-1f9a9a27da7b-kube-api-access-mbhtn\") pod \"078d2368-b084-4a49-a962-1f9a9a27da7b\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") "
Sep 10 23:03:25.893004 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.892989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-service-ca\") pod \"078d2368-b084-4a49-a962-1f9a9a27da7b\" (UID: \"078d2368-b084-4a49-a962-1f9a9a27da7b\") "
Sep 10 23:03:25.893279 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.893254    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-console-config" (OuterVolumeSpecName: "console-config") pod "078d2368-b084-4a49-a962-1f9a9a27da7b" (UID: "078d2368-b084-4a49-a962-1f9a9a27da7b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:25.893335 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.893274    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "078d2368-b084-4a49-a962-1f9a9a27da7b" (UID: "078d2368-b084-4a49-a962-1f9a9a27da7b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:25.893537 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.893516    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-service-ca" (OuterVolumeSpecName: "service-ca") pod "078d2368-b084-4a49-a962-1f9a9a27da7b" (UID: "078d2368-b084-4a49-a962-1f9a9a27da7b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:25.895260 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.895235    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078d2368-b084-4a49-a962-1f9a9a27da7b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "078d2368-b084-4a49-a962-1f9a9a27da7b" (UID: "078d2368-b084-4a49-a962-1f9a9a27da7b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:03:25.895260 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.895255    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078d2368-b084-4a49-a962-1f9a9a27da7b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "078d2368-b084-4a49-a962-1f9a9a27da7b" (UID: "078d2368-b084-4a49-a962-1f9a9a27da7b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:03:25.895386 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.895290    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078d2368-b084-4a49-a962-1f9a9a27da7b-kube-api-access-mbhtn" (OuterVolumeSpecName: "kube-api-access-mbhtn") pod "078d2368-b084-4a49-a962-1f9a9a27da7b" (UID: "078d2368-b084-4a49-a962-1f9a9a27da7b"). InnerVolumeSpecName "kube-api-access-mbhtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:03:25.994653 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.994620    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-oauth-serving-cert\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:03:25.994653 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.994648    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/078d2368-b084-4a49-a962-1f9a9a27da7b-console-serving-cert\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:03:25.994653 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.994657    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/078d2368-b084-4a49-a962-1f9a9a27da7b-console-oauth-config\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:03:25.994872 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.994665    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbhtn\" (UniqueName: \"kubernetes.io/projected/078d2368-b084-4a49-a962-1f9a9a27da7b-kube-api-access-mbhtn\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:03:25.994872 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.994674    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:03:25.994872 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:25.994683    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/078d2368-b084-4a49-a962-1f9a9a27da7b-console-config\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:03:26.509937 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:26.509899    2563 generic.go:356] "Generic (PLEG): container finished" podID="078d2368-b084-4a49-a962-1f9a9a27da7b" containerID="cc7654c176bc1298a63344c8eb60574f9e751ec50243e7de232f880eca7e403f" exitCode=0
Sep 10 23:03:26.510104 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:26.509947    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d467f779d-v2fr5"
Sep 10 23:03:26.510104 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:26.509990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d467f779d-v2fr5" event={"ID":"078d2368-b084-4a49-a962-1f9a9a27da7b","Type":"ContainerDied","Data":"cc7654c176bc1298a63344c8eb60574f9e751ec50243e7de232f880eca7e403f"}
Sep 10 23:03:26.510104 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:26.510049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d467f779d-v2fr5" event={"ID":"078d2368-b084-4a49-a962-1f9a9a27da7b","Type":"ContainerDied","Data":"9244b5b73a0b602d235812d83f510ec0a82770ad0bac8dac5761d55f8ef42436"}
Sep 10 23:03:26.510104 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:26.510070    2563 scope.go:117] "RemoveContainer" containerID="cc7654c176bc1298a63344c8eb60574f9e751ec50243e7de232f880eca7e403f"
Sep 10 23:03:26.518481 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:26.518463    2563 scope.go:117] "RemoveContainer" containerID="cc7654c176bc1298a63344c8eb60574f9e751ec50243e7de232f880eca7e403f"
Sep 10 23:03:26.518775 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:03:26.518757    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7654c176bc1298a63344c8eb60574f9e751ec50243e7de232f880eca7e403f\": container with ID starting with cc7654c176bc1298a63344c8eb60574f9e751ec50243e7de232f880eca7e403f not found: ID does not exist" containerID="cc7654c176bc1298a63344c8eb60574f9e751ec50243e7de232f880eca7e403f"
Sep 10 23:03:26.518822 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:26.518784    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7654c176bc1298a63344c8eb60574f9e751ec50243e7de232f880eca7e403f"} err="failed to get container status \"cc7654c176bc1298a63344c8eb60574f9e751ec50243e7de232f880eca7e403f\": rpc error: code = NotFound desc = could not find container \"cc7654c176bc1298a63344c8eb60574f9e751ec50243e7de232f880eca7e403f\": container with ID starting with cc7654c176bc1298a63344c8eb60574f9e751ec50243e7de232f880eca7e403f not found: ID does not exist"
Sep 10 23:03:26.530876 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:26.530856    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d467f779d-v2fr5"]
Sep 10 23:03:26.534418 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:26.534400    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d467f779d-v2fr5"]
Sep 10 23:03:27.812996 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:27.812964    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="078d2368-b084-4a49-a962-1f9a9a27da7b" path="/var/lib/kubelet/pods/078d2368-b084-4a49-a962-1f9a9a27da7b/volumes"
Sep 10 23:03:28.717439 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:28.717401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs\") pod \"network-metrics-daemon-v8sx8\" (UID: \"54f0cb94-e7d5-4b86-8602-dd8b46138dab\") " pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:03:28.719849 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:28.719820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f0cb94-e7d5-4b86-8602-dd8b46138dab-metrics-certs\") pod \"network-metrics-daemon-v8sx8\" (UID: \"54f0cb94-e7d5-4b86-8602-dd8b46138dab\") " pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:03:28.934936 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:28.934909    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-glltp\""
Sep 10 23:03:28.942856 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:28.942830    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v8sx8"
Sep 10 23:03:29.212349 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:29.212318    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v8sx8"]
Sep 10 23:03:29.215627 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:03:29.215598    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f0cb94_e7d5_4b86_8602_dd8b46138dab.slice/crio-f1f8631b7d4acea1447a76d3afa0ede0adf8973395fa89ceb92de1cb941131ac WatchSource:0}: Error finding container f1f8631b7d4acea1447a76d3afa0ede0adf8973395fa89ceb92de1cb941131ac: Status 404 returned error can't find the container with id f1f8631b7d4acea1447a76d3afa0ede0adf8973395fa89ceb92de1cb941131ac
Sep 10 23:03:29.523894 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:29.523858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v8sx8" event={"ID":"54f0cb94-e7d5-4b86-8602-dd8b46138dab","Type":"ContainerStarted","Data":"f1f8631b7d4acea1447a76d3afa0ede0adf8973395fa89ceb92de1cb941131ac"}
Sep 10 23:03:30.530115 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:30.530030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v8sx8" event={"ID":"54f0cb94-e7d5-4b86-8602-dd8b46138dab","Type":"ContainerStarted","Data":"9c8288afced8c3de0edd0e6ef3753e701ecdfd1f6416507a72f063132678ca55"}
Sep 10 23:03:30.530115 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:30.530100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v8sx8" event={"ID":"54f0cb94-e7d5-4b86-8602-dd8b46138dab","Type":"ContainerStarted","Data":"fdceac7b35903c1f97ab69cf0b3580aa2a74cbb599343527b29185f63f315c26"}
Sep 10 23:03:30.547382 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:30.547335    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v8sx8" podStartSLOduration=129.67075124 podStartE2EDuration="2m10.547320573s" totalImagesPullingTime="876.569333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:03:30.546040465 +0000 UTC m=+131.237617684" watchObservedRunningTime="2026-09-10 23:03:30.547320573 +0000 UTC m=+131.238897757"
Sep 10 23:03:31.223597 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.223556    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-666f85fc79-xmk8h" podUID="d44f82f8-998f-47b3-8470-a6a8e1c66a98" containerName="registry" containerID="cri-o://05fc27a7c37428220c3b7b81020a47cc545b048737d1834a2902ac21987e3f89" gracePeriod=30
Sep 10 23:03:31.519334 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.519310    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:03:31.534763 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.534733    2563 generic.go:356] "Generic (PLEG): container finished" podID="d44f82f8-998f-47b3-8470-a6a8e1c66a98" containerID="05fc27a7c37428220c3b7b81020a47cc545b048737d1834a2902ac21987e3f89" exitCode=0
Sep 10 23:03:31.535064 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.534772    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-666f85fc79-xmk8h"
Sep 10 23:03:31.535064 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.534816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666f85fc79-xmk8h" event={"ID":"d44f82f8-998f-47b3-8470-a6a8e1c66a98","Type":"ContainerDied","Data":"05fc27a7c37428220c3b7b81020a47cc545b048737d1834a2902ac21987e3f89"}
Sep 10 23:03:31.535064 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.534866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-666f85fc79-xmk8h" event={"ID":"d44f82f8-998f-47b3-8470-a6a8e1c66a98","Type":"ContainerDied","Data":"8a67a666f39182f9b2791a844375fb40a0b8cdfee20c9997db772f072a8c9f04"}
Sep 10 23:03:31.535064 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.534893    2563 scope.go:117] "RemoveContainer" containerID="05fc27a7c37428220c3b7b81020a47cc545b048737d1834a2902ac21987e3f89"
Sep 10 23:03:31.558573 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.558547    2563 scope.go:117] "RemoveContainer" containerID="05fc27a7c37428220c3b7b81020a47cc545b048737d1834a2902ac21987e3f89"
Sep 10 23:03:31.561895 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:03:31.561862    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05fc27a7c37428220c3b7b81020a47cc545b048737d1834a2902ac21987e3f89\": container with ID starting with 05fc27a7c37428220c3b7b81020a47cc545b048737d1834a2902ac21987e3f89 not found: ID does not exist" containerID="05fc27a7c37428220c3b7b81020a47cc545b048737d1834a2902ac21987e3f89"
Sep 10 23:03:31.562009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.561905    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05fc27a7c37428220c3b7b81020a47cc545b048737d1834a2902ac21987e3f89"} err="failed to get container status \"05fc27a7c37428220c3b7b81020a47cc545b048737d1834a2902ac21987e3f89\": rpc error: code = NotFound desc = could not find container \"05fc27a7c37428220c3b7b81020a47cc545b048737d1834a2902ac21987e3f89\": container with ID starting with 05fc27a7c37428220c3b7b81020a47cc545b048737d1834a2902ac21987e3f89 not found: ID does not exist"
Sep 10 23:03:31.643381 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.643348    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-certificates\") pod \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") "
Sep 10 23:03:31.643582 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.643399    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjwpf\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-kube-api-access-tjwpf\") pod \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") "
Sep 10 23:03:31.643582 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.643450    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls\") pod \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") "
Sep 10 23:03:31.643582 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.643518    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d44f82f8-998f-47b3-8470-a6a8e1c66a98-ca-trust-extracted\") pod \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") "
Sep 10 23:03:31.643582 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.643556    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d44f82f8-998f-47b3-8470-a6a8e1c66a98-trusted-ca\") pod \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") "
Sep 10 23:03:31.643804 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.643602    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d44f82f8-998f-47b3-8470-a6a8e1c66a98-image-registry-private-configuration\") pod \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") "
Sep 10 23:03:31.643804 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.643637    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d44f82f8-998f-47b3-8470-a6a8e1c66a98-installation-pull-secrets\") pod \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") "
Sep 10 23:03:31.643804 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.643675    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-bound-sa-token\") pod \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\" (UID: \"d44f82f8-998f-47b3-8470-a6a8e1c66a98\") "
Sep 10 23:03:31.643804 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.643744    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d44f82f8-998f-47b3-8470-a6a8e1c66a98" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:31.644015 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.643986    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-certificates\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:03:31.644015 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.643990    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44f82f8-998f-47b3-8470-a6a8e1c66a98-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d44f82f8-998f-47b3-8470-a6a8e1c66a98" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:03:31.646639 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.646598    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d44f82f8-998f-47b3-8470-a6a8e1c66a98" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:03:31.646871 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.646836    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d44f82f8-998f-47b3-8470-a6a8e1c66a98" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:03:31.646990 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.646861    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44f82f8-998f-47b3-8470-a6a8e1c66a98-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d44f82f8-998f-47b3-8470-a6a8e1c66a98" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:03:31.647233 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.647199    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-kube-api-access-tjwpf" (OuterVolumeSpecName: "kube-api-access-tjwpf") pod "d44f82f8-998f-47b3-8470-a6a8e1c66a98" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98"). InnerVolumeSpecName "kube-api-access-tjwpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:03:31.647433 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.647403    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44f82f8-998f-47b3-8470-a6a8e1c66a98-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d44f82f8-998f-47b3-8470-a6a8e1c66a98" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:03:31.661459 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.661424    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44f82f8-998f-47b3-8470-a6a8e1c66a98-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d44f82f8-998f-47b3-8470-a6a8e1c66a98" (UID: "d44f82f8-998f-47b3-8470-a6a8e1c66a98"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:03:31.745440 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.745362    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d44f82f8-998f-47b3-8470-a6a8e1c66a98-ca-trust-extracted\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:03:31.745440 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.745405    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d44f82f8-998f-47b3-8470-a6a8e1c66a98-trusted-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:03:31.745440 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.745421    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d44f82f8-998f-47b3-8470-a6a8e1c66a98-image-registry-private-configuration\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:03:31.745440 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.745437    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d44f82f8-998f-47b3-8470-a6a8e1c66a98-installation-pull-secrets\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:03:31.745726 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.745453    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-bound-sa-token\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:03:31.745726 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.745466    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjwpf\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-kube-api-access-tjwpf\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:03:31.745726 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.745479    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44f82f8-998f-47b3-8470-a6a8e1c66a98-registry-tls\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:03:31.851953 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.851916    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-666f85fc79-xmk8h"]
Sep 10 23:03:31.855422 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:31.855391    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-666f85fc79-xmk8h"]
Sep 10 23:03:33.813211 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:03:33.813171    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d44f82f8-998f-47b3-8470-a6a8e1c66a98" path="/var/lib/kubelet/pods/d44f82f8-998f-47b3-8470-a6a8e1c66a98/volumes"
Sep 10 23:04:02.324652 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:02.324606    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:02.340355 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:02.340329    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:02.674578 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:02.674510    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:16.091057 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.090979    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 23:04:16.091571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.091276    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="alertmanager" containerID="cri-o://3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609" gracePeriod=120
Sep 10 23:04:16.091571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.091343    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="kube-rbac-proxy-metric" containerID="cri-o://08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0" gracePeriod=120
Sep 10 23:04:16.091571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.091373    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="config-reloader" containerID="cri-o://b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9" gracePeriod=120
Sep 10 23:04:16.091571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.091391    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="prom-label-proxy" containerID="cri-o://39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a" gracePeriod=120
Sep 10 23:04:16.091571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.091425    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="kube-rbac-proxy" containerID="cri-o://c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b" gracePeriod=120
Sep 10 23:04:16.091571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.091354    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="kube-rbac-proxy-web" containerID="cri-o://41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee" gracePeriod=120
Sep 10 23:04:16.716439 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.716408    2563 generic.go:356] "Generic (PLEG): container finished" podID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerID="39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a" exitCode=0
Sep 10 23:04:16.716439 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.716434    2563 generic.go:356] "Generic (PLEG): container finished" podID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerID="c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b" exitCode=0
Sep 10 23:04:16.716439 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.716440    2563 generic.go:356] "Generic (PLEG): container finished" podID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerID="b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9" exitCode=0
Sep 10 23:04:16.716439 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.716446    2563 generic.go:356] "Generic (PLEG): container finished" podID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerID="3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609" exitCode=0
Sep 10 23:04:16.716720 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.716474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerDied","Data":"39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a"}
Sep 10 23:04:16.716720 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.716547    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerDied","Data":"c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b"}
Sep 10 23:04:16.716720 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.716577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerDied","Data":"b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9"}
Sep 10 23:04:16.716720 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:16.716602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerDied","Data":"3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609"}
Sep 10 23:04:17.358314 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.358282    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.442579 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.442487    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-main-tls\") pod \"984b8a72-f663-4249-8fe7-c5df654f8d42\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") "
Sep 10 23:04:17.442579 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.442550    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy\") pod \"984b8a72-f663-4249-8fe7-c5df654f8d42\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") "
Sep 10 23:04:17.442579 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.442573    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy-metric\") pod \"984b8a72-f663-4249-8fe7-c5df654f8d42\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") "
Sep 10 23:04:17.442844 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.442601    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy-web\") pod \"984b8a72-f663-4249-8fe7-c5df654f8d42\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") "
Sep 10 23:04:17.442844 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.442714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-cluster-tls-config\") pod \"984b8a72-f663-4249-8fe7-c5df654f8d42\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") "
Sep 10 23:04:17.442844 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.442798    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-web-config\") pod \"984b8a72-f663-4249-8fe7-c5df654f8d42\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") "
Sep 10 23:04:17.443045 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.442847    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-config-volume\") pod \"984b8a72-f663-4249-8fe7-c5df654f8d42\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") "
Sep 10 23:04:17.443045 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.442883    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-metrics-client-ca\") pod \"984b8a72-f663-4249-8fe7-c5df654f8d42\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") "
Sep 10 23:04:17.443045 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.442908    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/984b8a72-f663-4249-8fe7-c5df654f8d42-config-out\") pod \"984b8a72-f663-4249-8fe7-c5df654f8d42\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") "
Sep 10 23:04:17.443045 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.442932    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vf6x\" (UniqueName: \"kubernetes.io/projected/984b8a72-f663-4249-8fe7-c5df654f8d42-kube-api-access-6vf6x\") pod \"984b8a72-f663-4249-8fe7-c5df654f8d42\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") "
Sep 10 23:04:17.443045 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.442969    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/984b8a72-f663-4249-8fe7-c5df654f8d42-tls-assets\") pod \"984b8a72-f663-4249-8fe7-c5df654f8d42\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") "
Sep 10 23:04:17.443045 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.443040    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-trusted-ca-bundle\") pod \"984b8a72-f663-4249-8fe7-c5df654f8d42\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") "
Sep 10 23:04:17.443387 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.443071    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-main-db\") pod \"984b8a72-f663-4249-8fe7-c5df654f8d42\" (UID: \"984b8a72-f663-4249-8fe7-c5df654f8d42\") "
Sep 10 23:04:17.444075 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.443657    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "984b8a72-f663-4249-8fe7-c5df654f8d42" (UID: "984b8a72-f663-4249-8fe7-c5df654f8d42"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:04:17.444075 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.444044    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "984b8a72-f663-4249-8fe7-c5df654f8d42" (UID: "984b8a72-f663-4249-8fe7-c5df654f8d42"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:04:17.444247 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.444122    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "984b8a72-f663-4249-8fe7-c5df654f8d42" (UID: "984b8a72-f663-4249-8fe7-c5df654f8d42"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:04:17.446389 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.446220    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "984b8a72-f663-4249-8fe7-c5df654f8d42" (UID: "984b8a72-f663-4249-8fe7-c5df654f8d42"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:17.446389 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.446311    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "984b8a72-f663-4249-8fe7-c5df654f8d42" (UID: "984b8a72-f663-4249-8fe7-c5df654f8d42"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:17.446389 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.446363    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "984b8a72-f663-4249-8fe7-c5df654f8d42" (UID: "984b8a72-f663-4249-8fe7-c5df654f8d42"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:17.446631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.446564    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/984b8a72-f663-4249-8fe7-c5df654f8d42-config-out" (OuterVolumeSpecName: "config-out") pod "984b8a72-f663-4249-8fe7-c5df654f8d42" (UID: "984b8a72-f663-4249-8fe7-c5df654f8d42"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:04:17.447067 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.447026    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "984b8a72-f663-4249-8fe7-c5df654f8d42" (UID: "984b8a72-f663-4249-8fe7-c5df654f8d42"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:17.447275 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.447254    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984b8a72-f663-4249-8fe7-c5df654f8d42-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "984b8a72-f663-4249-8fe7-c5df654f8d42" (UID: "984b8a72-f663-4249-8fe7-c5df654f8d42"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:04:17.447398 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.447374    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-config-volume" (OuterVolumeSpecName: "config-volume") pod "984b8a72-f663-4249-8fe7-c5df654f8d42" (UID: "984b8a72-f663-4249-8fe7-c5df654f8d42"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:17.447652 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.447631    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984b8a72-f663-4249-8fe7-c5df654f8d42-kube-api-access-6vf6x" (OuterVolumeSpecName: "kube-api-access-6vf6x") pod "984b8a72-f663-4249-8fe7-c5df654f8d42" (UID: "984b8a72-f663-4249-8fe7-c5df654f8d42"). InnerVolumeSpecName "kube-api-access-6vf6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:04:17.451626 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.451608    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "984b8a72-f663-4249-8fe7-c5df654f8d42" (UID: "984b8a72-f663-4249-8fe7-c5df654f8d42"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:17.457708 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.457685    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-web-config" (OuterVolumeSpecName: "web-config") pod "984b8a72-f663-4249-8fe7-c5df654f8d42" (UID: "984b8a72-f663-4249-8fe7-c5df654f8d42"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:17.544073 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.544019    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-web-config\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:17.544073 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.544068    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:17.544073 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.544083    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-metrics-client-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:17.544333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.544096    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/984b8a72-f663-4249-8fe7-c5df654f8d42-config-out\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:17.544333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.544109    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vf6x\" (UniqueName: \"kubernetes.io/projected/984b8a72-f663-4249-8fe7-c5df654f8d42-kube-api-access-6vf6x\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:17.544333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.544120    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/984b8a72-f663-4249-8fe7-c5df654f8d42-tls-assets\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:17.544333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.544131    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:17.544333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.544143    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/984b8a72-f663-4249-8fe7-c5df654f8d42-alertmanager-main-db\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:17.544333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.544155    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-main-tls\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:17.544333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.544168    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:17.544333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.544181    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:17.544333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.544192    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:17.544333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.544205    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/984b8a72-f663-4249-8fe7-c5df654f8d42-cluster-tls-config\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:17.723225 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.723135    2563 generic.go:356] "Generic (PLEG): container finished" podID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerID="08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0" exitCode=0
Sep 10 23:04:17.723225 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.723166    2563 generic.go:356] "Generic (PLEG): container finished" podID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerID="41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee" exitCode=0
Sep 10 23:04:17.723225 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.723222    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.723443 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.723217    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerDied","Data":"08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0"}
Sep 10 23:04:17.723443 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.723399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerDied","Data":"41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee"}
Sep 10 23:04:17.723443 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.723428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"984b8a72-f663-4249-8fe7-c5df654f8d42","Type":"ContainerDied","Data":"134339db5c5d58ce9529aea84a25b98222ef7dc76a637bee26e4ba99f1c2d96b"}
Sep 10 23:04:17.723594 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.723459    2563 scope.go:117] "RemoveContainer" containerID="39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a"
Sep 10 23:04:17.731902 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.731879    2563 scope.go:117] "RemoveContainer" containerID="08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0"
Sep 10 23:04:17.739192 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.739172    2563 scope.go:117] "RemoveContainer" containerID="c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b"
Sep 10 23:04:17.746930 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.746910    2563 scope.go:117] "RemoveContainer" containerID="41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee"
Sep 10 23:04:17.747119 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.747098    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 23:04:17.752211 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.752187    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 23:04:17.754852 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.754838    2563 scope.go:117] "RemoveContainer" containerID="b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9"
Sep 10 23:04:17.761894 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.761872    2563 scope.go:117] "RemoveContainer" containerID="3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609"
Sep 10 23:04:17.769027 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.769007    2563 scope.go:117] "RemoveContainer" containerID="71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642"
Sep 10 23:04:17.778806 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.778786    2563 scope.go:117] "RemoveContainer" containerID="39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a"
Sep 10 23:04:17.779143 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:04:17.779117    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a\": container with ID starting with 39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a not found: ID does not exist" containerID="39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a"
Sep 10 23:04:17.779245 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779152    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a"} err="failed to get container status \"39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a\": rpc error: code = NotFound desc = could not find container \"39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a\": container with ID starting with 39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a not found: ID does not exist"
Sep 10 23:04:17.779245 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779171    2563 scope.go:117] "RemoveContainer" containerID="08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0"
Sep 10 23:04:17.779245 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779236    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 23:04:17.779504 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:04:17.779476    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0\": container with ID starting with 08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0 not found: ID does not exist" containerID="08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0"
Sep 10 23:04:17.779557 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779512    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0"} err="failed to get container status \"08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0\": rpc error: code = NotFound desc = could not find container \"08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0\": container with ID starting with 08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0 not found: ID does not exist"
Sep 10 23:04:17.779557 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779527    2563 scope.go:117] "RemoveContainer" containerID="c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779553    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="config-reloader"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779568    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="config-reloader"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779579    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="kube-rbac-proxy"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779594    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="kube-rbac-proxy"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779607    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="kube-rbac-proxy-web"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779613    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="kube-rbac-proxy-web"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779621    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="078d2368-b084-4a49-a962-1f9a9a27da7b" containerName="console"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779626    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="078d2368-b084-4a49-a962-1f9a9a27da7b" containerName="console"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779632    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="kube-rbac-proxy-metric"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779637    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="kube-rbac-proxy-metric"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779647    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44f82f8-998f-47b3-8470-a6a8e1c66a98" containerName="registry"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779652    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44f82f8-998f-47b3-8470-a6a8e1c66a98" containerName="registry"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779663    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="prom-label-proxy"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779670    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="prom-label-proxy"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779678    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="alertmanager"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779686    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="alertmanager"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779695    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="init-config-reloader"
Sep 10 23:04:17.779721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779700    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="init-config-reloader"
Sep 10 23:04:17.780295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779745    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="kube-rbac-proxy"
Sep 10 23:04:17.780295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779756    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="kube-rbac-proxy-metric"
Sep 10 23:04:17.780295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779766    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d44f82f8-998f-47b3-8470-a6a8e1c66a98" containerName="registry"
Sep 10 23:04:17.780295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779774    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="alertmanager"
Sep 10 23:04:17.780295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779784    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="kube-rbac-proxy-web"
Sep 10 23:04:17.780295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779794    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="config-reloader"
Sep 10 23:04:17.780295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779803    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" containerName="prom-label-proxy"
Sep 10 23:04:17.780295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779811    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="078d2368-b084-4a49-a962-1f9a9a27da7b" containerName="console"
Sep 10 23:04:17.780295 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:04:17.779815    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b\": container with ID starting with c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b not found: ID does not exist" containerID="c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b"
Sep 10 23:04:17.780295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779831    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b"} err="failed to get container status \"c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b\": rpc error: code = NotFound desc = could not find container \"c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b\": container with ID starting with c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b not found: ID does not exist"
Sep 10 23:04:17.780295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.779844    2563 scope.go:117] "RemoveContainer" containerID="41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee"
Sep 10 23:04:17.780295 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:04:17.780081    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee\": container with ID starting with 41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee not found: ID does not exist" containerID="41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee"
Sep 10 23:04:17.780295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.780111    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee"} err="failed to get container status \"41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee\": rpc error: code = NotFound desc = could not find container \"41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee\": container with ID starting with 41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee not found: ID does not exist"
Sep 10 23:04:17.780295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.780135    2563 scope.go:117] "RemoveContainer" containerID="b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9"
Sep 10 23:04:17.780810 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:04:17.780369    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9\": container with ID starting with b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9 not found: ID does not exist" containerID="b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9"
Sep 10 23:04:17.780810 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.780387    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9"} err="failed to get container status \"b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9\": rpc error: code = NotFound desc = could not find container \"b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9\": container with ID starting with b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9 not found: ID does not exist"
Sep 10 23:04:17.780810 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.780400    2563 scope.go:117] "RemoveContainer" containerID="3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609"
Sep 10 23:04:17.780810 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:04:17.780724    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609\": container with ID starting with 3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609 not found: ID does not exist" containerID="3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609"
Sep 10 23:04:17.780810 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.780756    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609"} err="failed to get container status \"3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609\": rpc error: code = NotFound desc = could not find container \"3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609\": container with ID starting with 3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609 not found: ID does not exist"
Sep 10 23:04:17.780810 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.780775    2563 scope.go:117] "RemoveContainer" containerID="71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642"
Sep 10 23:04:17.781054 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:04:17.781032    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642\": container with ID starting with 71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642 not found: ID does not exist" containerID="71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642"
Sep 10 23:04:17.781115 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.781062    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642"} err="failed to get container status \"71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642\": rpc error: code = NotFound desc = could not find container \"71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642\": container with ID starting with 71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642 not found: ID does not exist"
Sep 10 23:04:17.781115 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.781086    2563 scope.go:117] "RemoveContainer" containerID="39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a"
Sep 10 23:04:17.781327 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.781310    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a"} err="failed to get container status \"39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a\": rpc error: code = NotFound desc = could not find container \"39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a\": container with ID starting with 39fbc87375f809e8cf14a1bb2b2569f83ccf1d32420c3603a9148965282cb38a not found: ID does not exist"
Sep 10 23:04:17.781388 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.781330    2563 scope.go:117] "RemoveContainer" containerID="08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0"
Sep 10 23:04:17.781641 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.781622    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0"} err="failed to get container status \"08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0\": rpc error: code = NotFound desc = could not find container \"08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0\": container with ID starting with 08f2f0855c78d26b5628077d9dfb33bb2f6b41d8af610f971fc7f6012757fae0 not found: ID does not exist"
Sep 10 23:04:17.781685 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.781642    2563 scope.go:117] "RemoveContainer" containerID="c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b"
Sep 10 23:04:17.781867 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.781848    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b"} err="failed to get container status \"c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b\": rpc error: code = NotFound desc = could not find container \"c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b\": container with ID starting with c1eac4fb34801b99f323ca8cb882f3878320bbd273338b47e4f2e6ae13a8d56b not found: ID does not exist"
Sep 10 23:04:17.781914 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.781867    2563 scope.go:117] "RemoveContainer" containerID="41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee"
Sep 10 23:04:17.782092 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.782074    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee"} err="failed to get container status \"41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee\": rpc error: code = NotFound desc = could not find container \"41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee\": container with ID starting with 41afdb3d6cb2f55cd4eb971417d72887c440dbcc5d757134c08e1fa20da486ee not found: ID does not exist"
Sep 10 23:04:17.782159 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.782093    2563 scope.go:117] "RemoveContainer" containerID="b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9"
Sep 10 23:04:17.782320 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.782301    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9"} err="failed to get container status \"b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9\": rpc error: code = NotFound desc = could not find container \"b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9\": container with ID starting with b8e4f70b02c4b00e51c02d542b3ab1896515dfc267d26be57cb679538197adb9 not found: ID does not exist"
Sep 10 23:04:17.782389 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.782321    2563 scope.go:117] "RemoveContainer" containerID="3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609"
Sep 10 23:04:17.782596 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.782578    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609"} err="failed to get container status \"3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609\": rpc error: code = NotFound desc = could not find container \"3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609\": container with ID starting with 3d5eb5cb2bfaca22b3824d7aaeaa2132831d056db99aa3ad78ddfa8eb47a9609 not found: ID does not exist"
Sep 10 23:04:17.782647 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.782597    2563 scope.go:117] "RemoveContainer" containerID="71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642"
Sep 10 23:04:17.782801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.782783    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642"} err="failed to get container status \"71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642\": rpc error: code = NotFound desc = could not find container \"71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642\": container with ID starting with 71f3caa8bb506061e72653d0fe2db675f71136c5d339eee4b91328661ea16642 not found: ID does not exist"
Sep 10 23:04:17.784730 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.784714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.787281 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.787252    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 23:04:17.787281 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.787277    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 23:04:17.787433 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.787285    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nrrt5\""
Sep 10 23:04:17.787623 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.787608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 23:04:17.787686 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.787671    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 23:04:17.787786 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.787774    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 23:04:17.788015 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.788001    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 23:04:17.788066 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.788038    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 23:04:17.788109 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.788079    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 23:04:17.794240 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.794219    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 23:04:17.800010 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.799983    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 23:04:17.813329 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.813300    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="984b8a72-f663-4249-8fe7-c5df654f8d42" path="/var/lib/kubelet/pods/984b8a72-f663-4249-8fe7-c5df654f8d42/volumes"
Sep 10 23:04:17.947052 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.947008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.947052 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.947049    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.947255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.947067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.947255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.947115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.947255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.947157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-config-out\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.947255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.947177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-web-config\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.947255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.947222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-config-volume\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.947255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.947244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv4lz\" (UniqueName: \"kubernetes.io/projected/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-kube-api-access-cv4lz\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.947432 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.947279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.947432 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.947333    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.947432 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.947356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.947432 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.947384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:17.947432 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:17.947403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.048635 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.048555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-config-volume\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.048635 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.048589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv4lz\" (UniqueName: \"kubernetes.io/projected/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-kube-api-access-cv4lz\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.048635 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.048622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.048899 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.048656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.048899 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.048678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.048899 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.048706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.048899 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.048732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.048899 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.048770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.048899 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.048792    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.048899 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.048816    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.048899 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.048851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.048899 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.048876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-config-out\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.048899 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.048896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-web-config\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.049454 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.049427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.049859 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.049827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.052004 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.051760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-config-volume\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.052004 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.051760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.052004 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.051925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-web-config\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.052004 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.051927    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.052004 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.051947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.052294 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.052034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.052294 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.052165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.052397 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.052320    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.052439 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.052386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.053576 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.053557    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-config-out\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.059827 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.059807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv4lz\" (UniqueName: \"kubernetes.io/projected/8d54bd73-9bf1-41b3-b40e-cb3320ab429e-kube-api-access-cv4lz\") pod \"alertmanager-main-0\" (UID: \"8d54bd73-9bf1-41b3-b40e-cb3320ab429e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.094280 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.094248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 23:04:18.373649 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.373625    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 23:04:18.375677 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:04:18.375646    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d54bd73_9bf1_41b3_b40e_cb3320ab429e.slice/crio-eb47366e4c545fa81a82146ff2f8f37343cb41b3940d3e2c69d6a401e31a01b5 WatchSource:0}: Error finding container eb47366e4c545fa81a82146ff2f8f37343cb41b3940d3e2c69d6a401e31a01b5: Status 404 returned error can't find the container with id eb47366e4c545fa81a82146ff2f8f37343cb41b3940d3e2c69d6a401e31a01b5
Sep 10 23:04:18.728773 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.728684    2563 generic.go:356] "Generic (PLEG): container finished" podID="8d54bd73-9bf1-41b3-b40e-cb3320ab429e" containerID="854e7d4eba06fa96b9b0391430c97550a82ee2021049a7186268b51f77756b36" exitCode=0
Sep 10 23:04:18.728938 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.728768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d54bd73-9bf1-41b3-b40e-cb3320ab429e","Type":"ContainerDied","Data":"854e7d4eba06fa96b9b0391430c97550a82ee2021049a7186268b51f77756b36"}
Sep 10 23:04:18.728938 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:18.728841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d54bd73-9bf1-41b3-b40e-cb3320ab429e","Type":"ContainerStarted","Data":"eb47366e4c545fa81a82146ff2f8f37343cb41b3940d3e2c69d6a401e31a01b5"}
Sep 10 23:04:19.737245 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:19.737211    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d54bd73-9bf1-41b3-b40e-cb3320ab429e","Type":"ContainerStarted","Data":"4abf71f8fa91ef2aa2767958168dfcaad00a9250742fd58345e917b7129e6295"}
Sep 10 23:04:19.737562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:19.737317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d54bd73-9bf1-41b3-b40e-cb3320ab429e","Type":"ContainerStarted","Data":"0473c5024fa44cce83f586b4a54a91a23721ca699316ebd411c65fa7851d3f8f"}
Sep 10 23:04:19.737562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:19.737346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d54bd73-9bf1-41b3-b40e-cb3320ab429e","Type":"ContainerStarted","Data":"b118dc3619819426a913ba5f678f068f484871d8f2e3d17c233717ccb30c48bd"}
Sep 10 23:04:19.737562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:19.737378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d54bd73-9bf1-41b3-b40e-cb3320ab429e","Type":"ContainerStarted","Data":"767f783dd56b13f4105cce92444723db08540b13d2d9907a1937a6943b8f0974"}
Sep 10 23:04:19.737562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:19.737417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d54bd73-9bf1-41b3-b40e-cb3320ab429e","Type":"ContainerStarted","Data":"c9537785dfd7cc6bc83a27326909034330b99ad77fcd6b21b1cb8d1e8f2f53d6"}
Sep 10 23:04:19.737562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:19.737445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8d54bd73-9bf1-41b3-b40e-cb3320ab429e","Type":"ContainerStarted","Data":"ce3f8a7f2944e42ebc7ee1057879f7cd55b64b42df44dd08467dfd8192776f13"}
Sep 10 23:04:19.765235 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:19.765187    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.750791918 podStartE2EDuration="2.765172085s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.380167ms" isStatefulPod=false podCreationTimestamp="2026-09-10 23:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:04:19.762898653 +0000 UTC m=+180.454475858" watchObservedRunningTime="2026-09-10 23:04:19.765172085 +0000 UTC m=+180.456749268"
Sep 10 23:04:20.346777 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.346742    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:04:20.347040 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.347017    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="prometheus" containerID="cri-o://d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4" gracePeriod=600
Sep 10 23:04:20.347110 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.347035    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="kube-rbac-proxy" containerID="cri-o://c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c" gracePeriod=600
Sep 10 23:04:20.347110 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.347057    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="config-reloader" containerID="cri-o://d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e" gracePeriod=600
Sep 10 23:04:20.347110 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.347069    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="kube-rbac-proxy-web" containerID="cri-o://01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382" gracePeriod=600
Sep 10 23:04:20.347110 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.347103    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="kube-rbac-proxy-thanos" containerID="cri-o://79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b" gracePeriod=600
Sep 10 23:04:20.347334 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.347045    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="thanos-sidecar" containerID="cri-o://cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af" gracePeriod=600
Sep 10 23:04:20.582370 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.582347    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.672685 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.672600    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.672685 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.672650    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e4cd43c-288a-44a1-a70e-3e4be611b868-tls-assets\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.672685 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.672674    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-db\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.672951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.672711    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-grpc-tls\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.672951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.672737    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-kube-rbac-proxy\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.672951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.672769    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-thanos-prometheus-http-client-file\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.672951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.672801    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-kubelet-serving-ca-bundle\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.672951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.672823    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-0\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.672951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.672850    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e4cd43c-288a-44a1-a70e-3e4be611b868-config-out\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.672951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.672880    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-metrics-client-ca\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.672951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.672921    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.673346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.672958    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-web-config\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.673346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.672991    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-tls\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.673346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.673016    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-serving-certs-ca-bundle\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.673346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.673045    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-metrics-client-certs\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.673346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.673080    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-2\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.673346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.673112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctdt6\" (UniqueName: \"kubernetes.io/projected/6e4cd43c-288a-44a1-a70e-3e4be611b868-kube-api-access-ctdt6\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.673346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.673138    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-1\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.673346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.673167    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-trusted-ca-bundle\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.673346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.673204    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-config\") pod \"6e4cd43c-288a-44a1-a70e-3e4be611b868\" (UID: \"6e4cd43c-288a-44a1-a70e-3e4be611b868\") "
Sep 10 23:04:20.674217 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.674152    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:04:20.674561 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.674472    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:04:20.675060 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.675034    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:04:20.675157 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.675123    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:04:20.676380 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.675374    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:20.676380 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.675911    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:04:20.676380 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.676016    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:04:20.676380 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.676060    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:20.676380 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.676214    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4cd43c-288a-44a1-a70e-3e4be611b868-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:04:20.676380 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.676310    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:20.676380 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.676329    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4cd43c-288a-44a1-a70e-3e4be611b868-config-out" (OuterVolumeSpecName: "config-out") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:04:20.677122 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.676450    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:04:20.677122 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.676868    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:20.678053 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.678025    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:20.678404 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.678381    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-config" (OuterVolumeSpecName: "config") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:20.678557 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.678410    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:04:20.681864 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.681839    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:20.683153 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.683128    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4cd43c-288a-44a1-a70e-3e4be611b868-kube-api-access-ctdt6" (OuterVolumeSpecName: "kube-api-access-ctdt6") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "kube-api-access-ctdt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:04:20.683153 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.683138    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:20.690928 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.690909    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-web-config" (OuterVolumeSpecName: "web-config") pod "6e4cd43c-288a-44a1-a70e-3e4be611b868" (UID: "6e4cd43c-288a-44a1-a70e-3e4be611b868"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:04:20.744759 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.744727    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerID="79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b" exitCode=0
Sep 10 23:04:20.744759 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.744752    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerID="c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c" exitCode=0
Sep 10 23:04:20.744759 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.744758    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerID="01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382" exitCode=0
Sep 10 23:04:20.744759 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.744764    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerID="cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af" exitCode=0
Sep 10 23:04:20.745308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.744773    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerID="d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e" exitCode=0
Sep 10 23:04:20.745308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.744781    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerID="d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4" exitCode=0
Sep 10 23:04:20.745308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.744813    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.745308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.744809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerDied","Data":"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"}
Sep 10 23:04:20.745308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.745010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerDied","Data":"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"}
Sep 10 23:04:20.745308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.745067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerDied","Data":"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"}
Sep 10 23:04:20.745308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.745105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerDied","Data":"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"}
Sep 10 23:04:20.745308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.745145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerDied","Data":"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"}
Sep 10 23:04:20.745308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.745071    2563 scope.go:117] "RemoveContainer" containerID="79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"
Sep 10 23:04:20.745308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.745187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerDied","Data":"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"}
Sep 10 23:04:20.745308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.745224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6e4cd43c-288a-44a1-a70e-3e4be611b868","Type":"ContainerDied","Data":"b90d76223303a14f05617388eaae60eac9b70869e6e56a83177fff9fd3e0ab78"}
Sep 10 23:04:20.753095 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.753076    2563 scope.go:117] "RemoveContainer" containerID="c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"
Sep 10 23:04:20.759955 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.759937    2563 scope.go:117] "RemoveContainer" containerID="01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"
Sep 10 23:04:20.766299 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.766283    2563 scope.go:117] "RemoveContainer" containerID="cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"
Sep 10 23:04:20.768635 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.768613    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:04:20.773028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.773008    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:04:20.773918 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.773902    2563 scope.go:117] "RemoveContainer" containerID="d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"
Sep 10 23:04:20.774284 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774265    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774359 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774305    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-web-config\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774359 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774318    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774359 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774329    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774359 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774339    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-metrics-client-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774359 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774347    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774359 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774356    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctdt6\" (UniqueName: \"kubernetes.io/projected/6e4cd43c-288a-44a1-a70e-3e4be611b868-kube-api-access-ctdt6\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774599 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774369    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774599 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774384    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774599 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774397    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-config\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774599 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774410    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774599 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774422    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6e4cd43c-288a-44a1-a70e-3e4be611b868-tls-assets\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774599 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774436    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-db\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774599 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774445    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-grpc-tls\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774599 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774454    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-secret-kube-rbac-proxy\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774599 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774462    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6e4cd43c-288a-44a1-a70e-3e4be611b868-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774599 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774470    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774599 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774479    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774599 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774509    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6e4cd43c-288a-44a1-a70e-3e4be611b868-config-out\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.774599 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.774523    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e4cd43c-288a-44a1-a70e-3e4be611b868-configmap-metrics-client-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:04:20.780592 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.780569    2563 scope.go:117] "RemoveContainer" containerID="d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"
Sep 10 23:04:20.789636 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.789613    2563 scope.go:117] "RemoveContainer" containerID="6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"
Sep 10 23:04:20.796261 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.796246    2563 scope.go:117] "RemoveContainer" containerID="79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"
Sep 10 23:04:20.796578 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:04:20.796559    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b\": container with ID starting with 79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b not found: ID does not exist" containerID="79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"
Sep 10 23:04:20.796642 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.796586    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"} err="failed to get container status \"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b\": rpc error: code = NotFound desc = could not find container \"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b\": container with ID starting with 79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b not found: ID does not exist"
Sep 10 23:04:20.796642 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.796606    2563 scope.go:117] "RemoveContainer" containerID="c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"
Sep 10 23:04:20.796870 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:04:20.796850    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c\": container with ID starting with c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c not found: ID does not exist" containerID="c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"
Sep 10 23:04:20.796927 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.796879    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"} err="failed to get container status \"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c\": rpc error: code = NotFound desc = could not find container \"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c\": container with ID starting with c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c not found: ID does not exist"
Sep 10 23:04:20.796927 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.796904    2563 scope.go:117] "RemoveContainer" containerID="01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"
Sep 10 23:04:20.797149 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:04:20.797134    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382\": container with ID starting with 01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382 not found: ID does not exist" containerID="01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"
Sep 10 23:04:20.797190 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.797153    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"} err="failed to get container status \"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382\": rpc error: code = NotFound desc = could not find container \"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382\": container with ID starting with 01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382 not found: ID does not exist"
Sep 10 23:04:20.797190 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.797166    2563 scope.go:117] "RemoveContainer" containerID="cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"
Sep 10 23:04:20.797369 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:04:20.797353    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af\": container with ID starting with cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af not found: ID does not exist" containerID="cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"
Sep 10 23:04:20.797423 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.797375    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"} err="failed to get container status \"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af\": rpc error: code = NotFound desc = could not find container \"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af\": container with ID starting with cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af not found: ID does not exist"
Sep 10 23:04:20.797423 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.797394    2563 scope.go:117] "RemoveContainer" containerID="d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"
Sep 10 23:04:20.797633 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:04:20.797614    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e\": container with ID starting with d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e not found: ID does not exist" containerID="d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"
Sep 10 23:04:20.797668 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.797638    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"} err="failed to get container status \"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e\": rpc error: code = NotFound desc = could not find container \"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e\": container with ID starting with d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e not found: ID does not exist"
Sep 10 23:04:20.797668 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.797652    2563 scope.go:117] "RemoveContainer" containerID="d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"
Sep 10 23:04:20.797878 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:04:20.797863    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4\": container with ID starting with d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4 not found: ID does not exist" containerID="d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"
Sep 10 23:04:20.797916 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.797880    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"} err="failed to get container status \"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4\": rpc error: code = NotFound desc = could not find container \"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4\": container with ID starting with d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4 not found: ID does not exist"
Sep 10 23:04:20.797916 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.797895    2563 scope.go:117] "RemoveContainer" containerID="6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"
Sep 10 23:04:20.798098 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:04:20.798076    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd\": container with ID starting with 6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd not found: ID does not exist" containerID="6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"
Sep 10 23:04:20.798223 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.798103    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"} err="failed to get container status \"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd\": rpc error: code = NotFound desc = could not find container \"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd\": container with ID starting with 6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd not found: ID does not exist"
Sep 10 23:04:20.798223 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.798135    2563 scope.go:117] "RemoveContainer" containerID="79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"
Sep 10 23:04:20.798823 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.798683    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"} err="failed to get container status \"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b\": rpc error: code = NotFound desc = could not find container \"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b\": container with ID starting with 79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b not found: ID does not exist"
Sep 10 23:04:20.798823 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.798733    2563 scope.go:117] "RemoveContainer" containerID="c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"
Sep 10 23:04:20.799181 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.799154    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"} err="failed to get container status \"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c\": rpc error: code = NotFound desc = could not find container \"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c\": container with ID starting with c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c not found: ID does not exist"
Sep 10 23:04:20.799181 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.799180    2563 scope.go:117] "RemoveContainer" containerID="01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"
Sep 10 23:04:20.799471 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.799446    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"} err="failed to get container status \"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382\": rpc error: code = NotFound desc = could not find container \"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382\": container with ID starting with 01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382 not found: ID does not exist"
Sep 10 23:04:20.799471 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.799465    2563 scope.go:117] "RemoveContainer" containerID="cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"
Sep 10 23:04:20.799760 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.799742    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"} err="failed to get container status \"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af\": rpc error: code = NotFound desc = could not find container \"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af\": container with ID starting with cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af not found: ID does not exist"
Sep 10 23:04:20.799811 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.799761    2563 scope.go:117] "RemoveContainer" containerID="d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"
Sep 10 23:04:20.799976 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.799960    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"} err="failed to get container status \"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e\": rpc error: code = NotFound desc = could not find container \"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e\": container with ID starting with d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e not found: ID does not exist"
Sep 10 23:04:20.800023 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.799975    2563 scope.go:117] "RemoveContainer" containerID="d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"
Sep 10 23:04:20.800207 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800190    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"} err="failed to get container status \"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4\": rpc error: code = NotFound desc = could not find container \"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4\": container with ID starting with d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4 not found: ID does not exist"
Sep 10 23:04:20.800255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800209    2563 scope.go:117] "RemoveContainer" containerID="6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"
Sep 10 23:04:20.800434 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800417    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"} err="failed to get container status \"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd\": rpc error: code = NotFound desc = could not find container \"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd\": container with ID starting with 6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd not found: ID does not exist"
Sep 10 23:04:20.800514 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800434    2563 scope.go:117] "RemoveContainer" containerID="79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"
Sep 10 23:04:20.800566 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800526    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:04:20.800690 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800672    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"} err="failed to get container status \"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b\": rpc error: code = NotFound desc = could not find container \"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b\": container with ID starting with 79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b not found: ID does not exist"
Sep 10 23:04:20.800690 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800689    2563 scope.go:117] "RemoveContainer" containerID="c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"
Sep 10 23:04:20.800827 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800812    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="kube-rbac-proxy-web"
Sep 10 23:04:20.800861 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800831    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="kube-rbac-proxy-web"
Sep 10 23:04:20.800861 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800840    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="kube-rbac-proxy-thanos"
Sep 10 23:04:20.800861 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800845    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="kube-rbac-proxy-thanos"
Sep 10 23:04:20.800861 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800858    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="init-config-reloader"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800863    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="init-config-reloader"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800874    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="thanos-sidecar"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800882    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="thanos-sidecar"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800893    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="prometheus"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800898    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="prometheus"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800907    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="kube-rbac-proxy"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800912    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="kube-rbac-proxy"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800919    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="config-reloader"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800924    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="config-reloader"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800926    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"} err="failed to get container status \"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c\": rpc error: code = NotFound desc = could not find container \"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c\": container with ID starting with c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c not found: ID does not exist"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800949    2563 scope.go:117] "RemoveContainer" containerID="01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800969    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="thanos-sidecar"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800979    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="prometheus"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800989    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="kube-rbac-proxy-web"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.800999    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="kube-rbac-proxy-thanos"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.801007    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="config-reloader"
Sep 10 23:04:20.801009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.801017    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" containerName="kube-rbac-proxy"
Sep 10 23:04:20.801689 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.801190    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"} err="failed to get container status \"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382\": rpc error: code = NotFound desc = could not find container \"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382\": container with ID starting with 01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382 not found: ID does not exist"
Sep 10 23:04:20.801689 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.801211    2563 scope.go:117] "RemoveContainer" containerID="cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"
Sep 10 23:04:20.801689 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.801537    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"} err="failed to get container status \"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af\": rpc error: code = NotFound desc = could not find container \"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af\": container with ID starting with cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af not found: ID does not exist"
Sep 10 23:04:20.801689 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.801571    2563 scope.go:117] "RemoveContainer" containerID="d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"
Sep 10 23:04:20.801867 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.801848    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"} err="failed to get container status \"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e\": rpc error: code = NotFound desc = could not find container \"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e\": container with ID starting with d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e not found: ID does not exist"
Sep 10 23:04:20.801908 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.801867    2563 scope.go:117] "RemoveContainer" containerID="d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"
Sep 10 23:04:20.802104 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.802087    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"} err="failed to get container status \"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4\": rpc error: code = NotFound desc = could not find container \"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4\": container with ID starting with d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4 not found: ID does not exist"
Sep 10 23:04:20.802104 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.802103    2563 scope.go:117] "RemoveContainer" containerID="6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"
Sep 10 23:04:20.802323 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.802305    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"} err="failed to get container status \"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd\": rpc error: code = NotFound desc = could not find container \"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd\": container with ID starting with 6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd not found: ID does not exist"
Sep 10 23:04:20.802365 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.802323    2563 scope.go:117] "RemoveContainer" containerID="79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"
Sep 10 23:04:20.802541 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.802522    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"} err="failed to get container status \"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b\": rpc error: code = NotFound desc = could not find container \"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b\": container with ID starting with 79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b not found: ID does not exist"
Sep 10 23:04:20.802591 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.802541    2563 scope.go:117] "RemoveContainer" containerID="c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"
Sep 10 23:04:20.802799 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.802780    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"} err="failed to get container status \"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c\": rpc error: code = NotFound desc = could not find container \"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c\": container with ID starting with c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c not found: ID does not exist"
Sep 10 23:04:20.802848 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.802799    2563 scope.go:117] "RemoveContainer" containerID="01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"
Sep 10 23:04:20.803035 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.803018    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"} err="failed to get container status \"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382\": rpc error: code = NotFound desc = could not find container \"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382\": container with ID starting with 01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382 not found: ID does not exist"
Sep 10 23:04:20.803086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.803036    2563 scope.go:117] "RemoveContainer" containerID="cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"
Sep 10 23:04:20.803249 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.803233    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"} err="failed to get container status \"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af\": rpc error: code = NotFound desc = could not find container \"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af\": container with ID starting with cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af not found: ID does not exist"
Sep 10 23:04:20.803287 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.803249    2563 scope.go:117] "RemoveContainer" containerID="d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"
Sep 10 23:04:20.803467 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.803451    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"} err="failed to get container status \"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e\": rpc error: code = NotFound desc = could not find container \"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e\": container with ID starting with d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e not found: ID does not exist"
Sep 10 23:04:20.803523 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.803468    2563 scope.go:117] "RemoveContainer" containerID="d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"
Sep 10 23:04:20.803749 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.803726    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"} err="failed to get container status \"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4\": rpc error: code = NotFound desc = could not find container \"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4\": container with ID starting with d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4 not found: ID does not exist"
Sep 10 23:04:20.803749 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.803748    2563 scope.go:117] "RemoveContainer" containerID="6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"
Sep 10 23:04:20.804003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.803985    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"} err="failed to get container status \"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd\": rpc error: code = NotFound desc = could not find container \"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd\": container with ID starting with 6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd not found: ID does not exist"
Sep 10 23:04:20.804051 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.804003    2563 scope.go:117] "RemoveContainer" containerID="79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"
Sep 10 23:04:20.804218 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.804200    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"} err="failed to get container status \"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b\": rpc error: code = NotFound desc = could not find container \"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b\": container with ID starting with 79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b not found: ID does not exist"
Sep 10 23:04:20.804267 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.804218    2563 scope.go:117] "RemoveContainer" containerID="c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"
Sep 10 23:04:20.804435 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.804419    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"} err="failed to get container status \"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c\": rpc error: code = NotFound desc = could not find container \"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c\": container with ID starting with c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c not found: ID does not exist"
Sep 10 23:04:20.804435 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.804434    2563 scope.go:117] "RemoveContainer" containerID="01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"
Sep 10 23:04:20.804653 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.804637    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"} err="failed to get container status \"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382\": rpc error: code = NotFound desc = could not find container \"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382\": container with ID starting with 01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382 not found: ID does not exist"
Sep 10 23:04:20.804694 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.804653    2563 scope.go:117] "RemoveContainer" containerID="cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"
Sep 10 23:04:20.804858 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.804827    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"} err="failed to get container status \"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af\": rpc error: code = NotFound desc = could not find container \"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af\": container with ID starting with cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af not found: ID does not exist"
Sep 10 23:04:20.804897 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.804858    2563 scope.go:117] "RemoveContainer" containerID="d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"
Sep 10 23:04:20.805120 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.805101    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"} err="failed to get container status \"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e\": rpc error: code = NotFound desc = could not find container \"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e\": container with ID starting with d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e not found: ID does not exist"
Sep 10 23:04:20.805174 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.805121    2563 scope.go:117] "RemoveContainer" containerID="d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"
Sep 10 23:04:20.805353 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.805338    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"} err="failed to get container status \"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4\": rpc error: code = NotFound desc = could not find container \"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4\": container with ID starting with d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4 not found: ID does not exist"
Sep 10 23:04:20.805353 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.805352    2563 scope.go:117] "RemoveContainer" containerID="6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"
Sep 10 23:04:20.805581 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.805562    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"} err="failed to get container status \"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd\": rpc error: code = NotFound desc = could not find container \"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd\": container with ID starting with 6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd not found: ID does not exist"
Sep 10 23:04:20.805620 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.805582    2563 scope.go:117] "RemoveContainer" containerID="79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"
Sep 10 23:04:20.805808 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.805785    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b"} err="failed to get container status \"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b\": rpc error: code = NotFound desc = could not find container \"79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b\": container with ID starting with 79da62022634b9182d2ceeca3f6281de4d87690b44f032c96e5ad8c9c1401b7b not found: ID does not exist"
Sep 10 23:04:20.805860 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.805810    2563 scope.go:117] "RemoveContainer" containerID="c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"
Sep 10 23:04:20.806060 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.806046    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c"} err="failed to get container status \"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c\": rpc error: code = NotFound desc = could not find container \"c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c\": container with ID starting with c7cf05505b554a1d343f3704f543c13f11a5cad3223ac12d8c744674b3343d8c not found: ID does not exist"
Sep 10 23:04:20.806104 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.806061    2563 scope.go:117] "RemoveContainer" containerID="01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"
Sep 10 23:04:20.806284 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.806264    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382"} err="failed to get container status \"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382\": rpc error: code = NotFound desc = could not find container \"01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382\": container with ID starting with 01f3f8c8271a3f45fac9ab08abd7e3272f0fc472e9d0db0ad5fdb9ffbbb7e382 not found: ID does not exist"
Sep 10 23:04:20.806356 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.806285    2563 scope.go:117] "RemoveContainer" containerID="cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"
Sep 10 23:04:20.806567 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.806553    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.806653 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.806556    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af"} err="failed to get container status \"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af\": rpc error: code = NotFound desc = could not find container \"cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af\": container with ID starting with cda0818482b021effd7d1662381d89e49e8e41506fd906954a79143f148c20af not found: ID does not exist"
Sep 10 23:04:20.806743 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.806654    2563 scope.go:117] "RemoveContainer" containerID="d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"
Sep 10 23:04:20.807157 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.807135    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e"} err="failed to get container status \"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e\": rpc error: code = NotFound desc = could not find container \"d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e\": container with ID starting with d8acd9e234ce9aed8ba5cc28deddf647192ab1e4f203a2bf1a15be9cf0bc054e not found: ID does not exist"
Sep 10 23:04:20.807157 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.807157    2563 scope.go:117] "RemoveContainer" containerID="d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"
Sep 10 23:04:20.807420 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.807391    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4"} err="failed to get container status \"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4\": rpc error: code = NotFound desc = could not find container \"d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4\": container with ID starting with d47c2fb8443096d0414effbb98fbf2252ac9ae62d9f17fe6c466fd58e0ff96d4 not found: ID does not exist"
Sep 10 23:04:20.807512 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.807422    2563 scope.go:117] "RemoveContainer" containerID="6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"
Sep 10 23:04:20.807700 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.807682    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd"} err="failed to get container status \"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd\": rpc error: code = NotFound desc = could not find container \"6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd\": container with ID starting with 6cac1cc36715326fbb5f7f6e4b2cb095d407d232594c472eddbb148b72ecaedd not found: ID does not exist"
Sep 10 23:04:20.809132 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.809092    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 23:04:20.809132 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.809104    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 23:04:20.809247 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.809227    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 23:04:20.809421 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.809406    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 23:04:20.809505 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.809405    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7z8pw\""
Sep 10 23:04:20.809603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.809558    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 23:04:20.809748 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.809733    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 23:04:20.809821 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.809763    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 23:04:20.809821 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.809808    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 23:04:20.809929 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.809831    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 23:04:20.809929 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.809836    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-12ks42kfv2ac3\""
Sep 10 23:04:20.809929 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.809766    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 23:04:20.810130 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.810111    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 23:04:20.810229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.810159    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 23:04:20.810277 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.810158    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 23:04:20.812418 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.812382    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 23:04:20.815132 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.815113    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 23:04:20.818443 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.818422    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:04:20.976696 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.976612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.976696 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.976646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.976696 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.976667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.976696 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.976687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.976743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.976771    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.976803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.976841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t44s\" (UniqueName: \"kubernetes.io/projected/5ccdee23-a231-44be-a1d4-55ff61a2afc4-kube-api-access-7t44s\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.976869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.976885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.976904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.976941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.976962    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.976994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-config\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.977019    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ccdee23-a231-44be-a1d4-55ff61a2afc4-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977362 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.977038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ccdee23-a231-44be-a1d4-55ff61a2afc4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977362 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.977056    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977362 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.977076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977362 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.977096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:20.977362 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:20.977115    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.077611 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.077577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.077762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.077643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-config\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.077762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.077675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ccdee23-a231-44be-a1d4-55ff61a2afc4-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.077762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.077703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ccdee23-a231-44be-a1d4-55ff61a2afc4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.077762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.077732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.077762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.077757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.078028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.077790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.078028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.077814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.078028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.077843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.078028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.077876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.078028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.077906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.078028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.077937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.078028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.077962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.078028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.077988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.078028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.078015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.078444 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.078045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t44s\" (UniqueName: \"kubernetes.io/projected/5ccdee23-a231-44be-a1d4-55ff61a2afc4-kube-api-access-7t44s\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.078444 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.078081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.078444 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.078109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.078444 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.078139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.078444 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.078170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.080896 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.079465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.080896 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.080133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.080896 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.080731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.081130 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.081035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.081292 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.081270    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-config\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.081391 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.081279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.081456 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.081443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.081933 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.081614    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ccdee23-a231-44be-a1d4-55ff61a2afc4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.081933 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.081620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.081933 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.081882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.082128 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.081981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.082128 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.082028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-web-config\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.082232 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.082168    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.082287 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.082240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5ccdee23-a231-44be-a1d4-55ff61a2afc4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.082524 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.082484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.083040 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.083017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ccdee23-a231-44be-a1d4-55ff61a2afc4-config-out\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.083355 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.083339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.083825 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.083807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.083933 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.083917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5ccdee23-a231-44be-a1d4-55ff61a2afc4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.087210 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.087194    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t44s\" (UniqueName: \"kubernetes.io/projected/5ccdee23-a231-44be-a1d4-55ff61a2afc4-kube-api-access-7t44s\") pod \"prometheus-k8s-0\" (UID: \"5ccdee23-a231-44be-a1d4-55ff61a2afc4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.116064 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.116038    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:04:21.420685 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.420654    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 23:04:21.420913 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:04:21.420888    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccdee23_a231_44be_a1d4_55ff61a2afc4.slice/crio-97bf58d1f4222c7532c56ca365c396b24e0173ba2661fb8f0b4789c192113264 WatchSource:0}: Error finding container 97bf58d1f4222c7532c56ca365c396b24e0173ba2661fb8f0b4789c192113264: Status 404 returned error can't find the container with id 97bf58d1f4222c7532c56ca365c396b24e0173ba2661fb8f0b4789c192113264
Sep 10 23:04:21.750798 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.750716    2563 generic.go:356] "Generic (PLEG): container finished" podID="5ccdee23-a231-44be-a1d4-55ff61a2afc4" containerID="3f901b520de5f2f34d2afc61c1275bc9a7b2ba15b939b851126761f1628d5e35" exitCode=0
Sep 10 23:04:21.751317 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.750810    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ccdee23-a231-44be-a1d4-55ff61a2afc4","Type":"ContainerDied","Data":"3f901b520de5f2f34d2afc61c1275bc9a7b2ba15b939b851126761f1628d5e35"}
Sep 10 23:04:21.751317 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.750879    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ccdee23-a231-44be-a1d4-55ff61a2afc4","Type":"ContainerStarted","Data":"97bf58d1f4222c7532c56ca365c396b24e0173ba2661fb8f0b4789c192113264"}
Sep 10 23:04:21.813873 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:21.813841    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e4cd43c-288a-44a1-a70e-3e4be611b868" path="/var/lib/kubelet/pods/6e4cd43c-288a-44a1-a70e-3e4be611b868/volumes"
Sep 10 23:04:22.759459 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:22.759374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ccdee23-a231-44be-a1d4-55ff61a2afc4","Type":"ContainerStarted","Data":"b4bb9ff073721ce19bb4d68584655f08240fb40afbe802ca61cc864daaada1e0"}
Sep 10 23:04:22.759839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:22.759511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ccdee23-a231-44be-a1d4-55ff61a2afc4","Type":"ContainerStarted","Data":"5c648785ad0de0e3c7852f94554f77b59831941647f6c3b0226a934ed3b2cc58"}
Sep 10 23:04:22.759839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:22.759557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ccdee23-a231-44be-a1d4-55ff61a2afc4","Type":"ContainerStarted","Data":"e55c4d292a75f3c2590bfd438c37dc25a462ae0a2e052bf15ca497602fe0e6af"}
Sep 10 23:04:22.759839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:22.759603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ccdee23-a231-44be-a1d4-55ff61a2afc4","Type":"ContainerStarted","Data":"43f7f9305731a58a88f2534dcc11760762bfd73d912e685b4712c455cf2f2679"}
Sep 10 23:04:22.759839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:22.759640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ccdee23-a231-44be-a1d4-55ff61a2afc4","Type":"ContainerStarted","Data":"ade56c317e72809dadf5d253a38308a6c0fa980796b2e23d175117689b7282fd"}
Sep 10 23:04:22.759839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:22.759675    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5ccdee23-a231-44be-a1d4-55ff61a2afc4","Type":"ContainerStarted","Data":"882aa54ffafb3153bd7c05db8058f8afebf5ef6fb7fad9c69eb7d314a4a6c322"}
Sep 10 23:04:22.807511 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:22.807423    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.786236943 podStartE2EDuration="2.807401372s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.164429ms" isStatefulPod=false podCreationTimestamp="2026-09-10 23:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:04:22.805030986 +0000 UTC m=+183.496608218" watchObservedRunningTime="2026-09-10 23:04:22.807401372 +0000 UTC m=+183.498978558"
Sep 10 23:04:26.116594 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:04:26.116548    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:05:21.116820 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:05:21.116784    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:05:21.132035 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:05:21.132011    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:05:22.011593 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:05:22.011565    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 23:06:03.369051 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.369007    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"]
Sep 10 23:06:03.372579 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.372559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:03.377970 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.377948    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 23:06:03.378099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.378066    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-m282x\""
Sep 10 23:06:03.378099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.378084    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 23:06:03.379177 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.378833    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 23:06:03.379177 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.379000    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 23:06:03.379177 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.379017    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 23:06:03.387294 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.387269    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"]
Sep 10 23:06:03.524359 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.524326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a8c3c281-3f33-4c3c-9ac6-99ce91f7556a-manager-config\") pod \"lws-controller-manager-559dcbdd76-4h95d\" (UID: \"a8c3c281-3f33-4c3c-9ac6-99ce91f7556a\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:03.524608 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.524380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6fn2\" (UniqueName: \"kubernetes.io/projected/a8c3c281-3f33-4c3c-9ac6-99ce91f7556a-kube-api-access-v6fn2\") pod \"lws-controller-manager-559dcbdd76-4h95d\" (UID: \"a8c3c281-3f33-4c3c-9ac6-99ce91f7556a\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:03.524608 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.524471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a8c3c281-3f33-4c3c-9ac6-99ce91f7556a-metrics-cert\") pod \"lws-controller-manager-559dcbdd76-4h95d\" (UID: \"a8c3c281-3f33-4c3c-9ac6-99ce91f7556a\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:03.524608 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.524519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8c3c281-3f33-4c3c-9ac6-99ce91f7556a-cert\") pod \"lws-controller-manager-559dcbdd76-4h95d\" (UID: \"a8c3c281-3f33-4c3c-9ac6-99ce91f7556a\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:03.625720 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.625643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6fn2\" (UniqueName: \"kubernetes.io/projected/a8c3c281-3f33-4c3c-9ac6-99ce91f7556a-kube-api-access-v6fn2\") pod \"lws-controller-manager-559dcbdd76-4h95d\" (UID: \"a8c3c281-3f33-4c3c-9ac6-99ce91f7556a\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:03.625720 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.625693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a8c3c281-3f33-4c3c-9ac6-99ce91f7556a-metrics-cert\") pod \"lws-controller-manager-559dcbdd76-4h95d\" (UID: \"a8c3c281-3f33-4c3c-9ac6-99ce91f7556a\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:03.625720 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.625714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8c3c281-3f33-4c3c-9ac6-99ce91f7556a-cert\") pod \"lws-controller-manager-559dcbdd76-4h95d\" (UID: \"a8c3c281-3f33-4c3c-9ac6-99ce91f7556a\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:03.625967 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.625747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a8c3c281-3f33-4c3c-9ac6-99ce91f7556a-manager-config\") pod \"lws-controller-manager-559dcbdd76-4h95d\" (UID: \"a8c3c281-3f33-4c3c-9ac6-99ce91f7556a\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:03.626336 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.626316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a8c3c281-3f33-4c3c-9ac6-99ce91f7556a-manager-config\") pod \"lws-controller-manager-559dcbdd76-4h95d\" (UID: \"a8c3c281-3f33-4c3c-9ac6-99ce91f7556a\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:03.628216 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.628192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8c3c281-3f33-4c3c-9ac6-99ce91f7556a-cert\") pod \"lws-controller-manager-559dcbdd76-4h95d\" (UID: \"a8c3c281-3f33-4c3c-9ac6-99ce91f7556a\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:03.628299 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.628276    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a8c3c281-3f33-4c3c-9ac6-99ce91f7556a-metrics-cert\") pod \"lws-controller-manager-559dcbdd76-4h95d\" (UID: \"a8c3c281-3f33-4c3c-9ac6-99ce91f7556a\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:03.636641 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.636619    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6fn2\" (UniqueName: \"kubernetes.io/projected/a8c3c281-3f33-4c3c-9ac6-99ce91f7556a-kube-api-access-v6fn2\") pod \"lws-controller-manager-559dcbdd76-4h95d\" (UID: \"a8c3c281-3f33-4c3c-9ac6-99ce91f7556a\") " pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:03.682097 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.682070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:03.958313 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:03.958223    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"]
Sep 10 23:06:03.964985 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:06:03.964941    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c3c281_3f33_4c3c_9ac6_99ce91f7556a.slice/crio-ccfb8bbceb866da07d84552f9f6f666576d503ee03c5f60fe788a96aa49f09ef WatchSource:0}: Error finding container ccfb8bbceb866da07d84552f9f6f666576d503ee03c5f60fe788a96aa49f09ef: Status 404 returned error can't find the container with id ccfb8bbceb866da07d84552f9f6f666576d503ee03c5f60fe788a96aa49f09ef
Sep 10 23:06:04.113351 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.113313    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"]
Sep 10 23:06:04.118123 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.118099    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.120835 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.120812    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 23:06:04.121352 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.121321    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-z7b9q\""
Sep 10 23:06:04.121456 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.121360    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 23:06:04.121456 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.121408    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 23:06:04.121602 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.121413    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 23:06:04.138332 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.138305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"]
Sep 10 23:06:04.159049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.159019    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d" event={"ID":"a8c3c281-3f33-4c3c-9ac6-99ce91f7556a","Type":"ContainerStarted","Data":"ccfb8bbceb866da07d84552f9f6f666576d503ee03c5f60fe788a96aa49f09ef"}
Sep 10 23:06:04.232704 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.232621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.232704 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.232678    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.232874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.232708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtprp\" (UniqueName: \"kubernetes.io/projected/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-kube-api-access-dtprp\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.232874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.232728    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-manifests\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.232874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.232747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-tmp\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.333471 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.333432    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.333678 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.333509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.333678 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.333535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtprp\" (UniqueName: \"kubernetes.io/projected/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-kube-api-access-dtprp\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.333678 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.333554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-manifests\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.333678 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.333575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-tmp\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.334044 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.334023    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-manifests\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.335794 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.335775    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-tmp\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.335928 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.335911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.336028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.336013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-webhook-cert\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.347775 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.347757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtprp\" (UniqueName: \"kubernetes.io/projected/8a9fe207-ecdb-494e-8bce-4fbd147f8ef6-kube-api-access-dtprp\") pod \"opendatahub-operator-controller-manager-f5b5b6476-7qp5k\" (UID: \"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.430314 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.430283    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:04.706839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:04.706703    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"]
Sep 10 23:06:04.709208 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:06:04.709178    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9fe207_ecdb_494e_8bce_4fbd147f8ef6.slice/crio-4dadbbac5e0f43c9962c9c8e9fa49def49b3e855b10cb45596772621962aff40 WatchSource:0}: Error finding container 4dadbbac5e0f43c9962c9c8e9fa49def49b3e855b10cb45596772621962aff40: Status 404 returned error can't find the container with id 4dadbbac5e0f43c9962c9c8e9fa49def49b3e855b10cb45596772621962aff40
Sep 10 23:06:05.167464 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:05.167401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k" event={"ID":"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6","Type":"ContainerStarted","Data":"4dadbbac5e0f43c9962c9c8e9fa49def49b3e855b10cb45596772621962aff40"}
Sep 10 23:06:09.191256 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:09.191217    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d" event={"ID":"a8c3c281-3f33-4c3c-9ac6-99ce91f7556a","Type":"ContainerStarted","Data":"1903ed34bf47a998a87e6a58780cca1fb258fc40a5e24298f653fa6e76d88f3a"}
Sep 10 23:06:09.191746 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:09.191349    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:09.192837 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:09.192816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k" event={"ID":"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6","Type":"ContainerStarted","Data":"5552699398c6f9ce7ff417fc9195c4c3a51f10f3d78c16cb1f30ebd35ec11fce"}
Sep 10 23:06:09.209901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:09.209849    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d" podStartSLOduration=1.961894279 podStartE2EDuration="6.209837803s" totalImagesPullingTime="4.247943524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:09.209245575 +0000 UTC m=+289.900822771" watchObservedRunningTime="2026-09-10 23:06:09.209837803 +0000 UTC m=+289.901414987"
Sep 10 23:06:10.201245 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:10.201212    2563 generic.go:356] "Generic (PLEG): container finished" podID="8a9fe207-ecdb-494e-8bce-4fbd147f8ef6" containerID="5552699398c6f9ce7ff417fc9195c4c3a51f10f3d78c16cb1f30ebd35ec11fce" exitCode=0
Sep 10 23:06:10.201697 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:10.201306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k" event={"ID":"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6","Type":"ContainerDied","Data":"5552699398c6f9ce7ff417fc9195c4c3a51f10f3d78c16cb1f30ebd35ec11fce"}
Sep 10 23:06:11.208162 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:11.208122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k" event={"ID":"8a9fe207-ecdb-494e-8bce-4fbd147f8ef6","Type":"ContainerStarted","Data":"a93f9088ae1a34be0f48bfd490522f12711f82ea289b775244be19b8bdf3a669"}
Sep 10 23:06:11.208656 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:11.208236    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:11.232384 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:11.232336    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k" podStartSLOduration=2.18190569 podStartE2EDuration="7.23231956s" totalImagesPullingTime="3.921999863s" totalInitContainerRuntime="1.128414007s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:11.232081042 +0000 UTC m=+291.923658250" watchObservedRunningTime="2026-09-10 23:06:11.23231956 +0000 UTC m=+291.923896745"
Sep 10 23:06:19.739825 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:19.739798    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:06:19.740752 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:19.740721    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:06:19.747308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:19.747284    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 23:06:20.203979 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:20.203949    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-559dcbdd76-4h95d"
Sep 10 23:06:22.215180 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:22.215148    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f5b5b6476-7qp5k"
Sep 10 23:06:24.216821 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.216788    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr"]
Sep 10 23:06:24.222449 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.222428    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr"
Sep 10 23:06:24.224764 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.224744    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-d85ct\""
Sep 10 23:06:24.225571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.225553    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 23:06:24.225671 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.225590    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 23:06:24.232872 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.232847    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr"]
Sep 10 23:06:24.301382 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.301345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a46819a0-33b2-4794-8112-139132fb0102-tmp\") pod \"kube-auth-proxy-5d6c69d6b7-stgvr\" (UID: \"a46819a0-33b2-4794-8112-139132fb0102\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr"
Sep 10 23:06:24.301382 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.301384    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52sr\" (UniqueName: \"kubernetes.io/projected/a46819a0-33b2-4794-8112-139132fb0102-kube-api-access-g52sr\") pod \"kube-auth-proxy-5d6c69d6b7-stgvr\" (UID: \"a46819a0-33b2-4794-8112-139132fb0102\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr"
Sep 10 23:06:24.301616 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.301412    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46819a0-33b2-4794-8112-139132fb0102-tls-certs\") pod \"kube-auth-proxy-5d6c69d6b7-stgvr\" (UID: \"a46819a0-33b2-4794-8112-139132fb0102\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr"
Sep 10 23:06:24.401874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.401838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46819a0-33b2-4794-8112-139132fb0102-tls-certs\") pod \"kube-auth-proxy-5d6c69d6b7-stgvr\" (UID: \"a46819a0-33b2-4794-8112-139132fb0102\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr"
Sep 10 23:06:24.401963 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.401939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a46819a0-33b2-4794-8112-139132fb0102-tmp\") pod \"kube-auth-proxy-5d6c69d6b7-stgvr\" (UID: \"a46819a0-33b2-4794-8112-139132fb0102\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr"
Sep 10 23:06:24.402003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.401962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52sr\" (UniqueName: \"kubernetes.io/projected/a46819a0-33b2-4794-8112-139132fb0102-kube-api-access-g52sr\") pod \"kube-auth-proxy-5d6c69d6b7-stgvr\" (UID: \"a46819a0-33b2-4794-8112-139132fb0102\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr"
Sep 10 23:06:24.404260 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.404230    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a46819a0-33b2-4794-8112-139132fb0102-tmp\") pod \"kube-auth-proxy-5d6c69d6b7-stgvr\" (UID: \"a46819a0-33b2-4794-8112-139132fb0102\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr"
Sep 10 23:06:24.404415 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.404399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46819a0-33b2-4794-8112-139132fb0102-tls-certs\") pod \"kube-auth-proxy-5d6c69d6b7-stgvr\" (UID: \"a46819a0-33b2-4794-8112-139132fb0102\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr"
Sep 10 23:06:24.411806 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.411781    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52sr\" (UniqueName: \"kubernetes.io/projected/a46819a0-33b2-4794-8112-139132fb0102-kube-api-access-g52sr\") pod \"kube-auth-proxy-5d6c69d6b7-stgvr\" (UID: \"a46819a0-33b2-4794-8112-139132fb0102\") " pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr"
Sep 10 23:06:24.532883 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.532795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr"
Sep 10 23:06:24.809460 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.809429    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr"]
Sep 10 23:06:24.812164 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:06:24.812135    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46819a0_33b2_4794_8112_139132fb0102.slice/crio-c22c1919eba58a01fc2814143d2b48c831decca0f56123629992726927488d38 WatchSource:0}: Error finding container c22c1919eba58a01fc2814143d2b48c831decca0f56123629992726927488d38: Status 404 returned error can't find the container with id c22c1919eba58a01fc2814143d2b48c831decca0f56123629992726927488d38
Sep 10 23:06:24.813988 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:24.813972    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:06:25.270377 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:25.270300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr" event={"ID":"a46819a0-33b2-4794-8112-139132fb0102","Type":"ContainerStarted","Data":"c22c1919eba58a01fc2814143d2b48c831decca0f56123629992726927488d38"}
Sep 10 23:06:28.286116 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:28.286070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr" event={"ID":"a46819a0-33b2-4794-8112-139132fb0102","Type":"ContainerStarted","Data":"45057b83264a3ead5b16f2a47207dcc525802aab9adce6c08a6630b82da451d6"}
Sep 10 23:06:28.304529 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:06:28.304464    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d6c69d6b7-stgvr" podStartSLOduration=1.6054541580000001 podStartE2EDuration="4.304447145s" totalImagesPullingTime="2.698992987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:28.303019001 +0000 UTC m=+308.994596219" watchObservedRunningTime="2026-09-10 23:06:28.304447145 +0000 UTC m=+308.996024329"
Sep 10 23:08:22.301242 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:22.301206    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"]
Sep 10 23:08:22.304563 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:22.304545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"
Sep 10 23:08:22.308795 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:22.308771    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 23:08:22.308933 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:22.308912    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 23:08:22.309020 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:22.309005    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-thk95\""
Sep 10 23:08:22.318831 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:22.318803    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"]
Sep 10 23:08:22.400646 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:22.400612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdb9g\" (UniqueName: \"kubernetes.io/projected/3537a2cf-bd5a-41e3-8f8e-01b066e279d3-kube-api-access-cdb9g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dnr2j\" (UID: \"3537a2cf-bd5a-41e3-8f8e-01b066e279d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"
Sep 10 23:08:22.400843 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:22.400684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3537a2cf-bd5a-41e3-8f8e-01b066e279d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dnr2j\" (UID: \"3537a2cf-bd5a-41e3-8f8e-01b066e279d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"
Sep 10 23:08:22.501794 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:22.501752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3537a2cf-bd5a-41e3-8f8e-01b066e279d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dnr2j\" (UID: \"3537a2cf-bd5a-41e3-8f8e-01b066e279d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"
Sep 10 23:08:22.501970 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:22.501862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdb9g\" (UniqueName: \"kubernetes.io/projected/3537a2cf-bd5a-41e3-8f8e-01b066e279d3-kube-api-access-cdb9g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dnr2j\" (UID: \"3537a2cf-bd5a-41e3-8f8e-01b066e279d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"
Sep 10 23:08:22.502134 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:22.502112    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3537a2cf-bd5a-41e3-8f8e-01b066e279d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dnr2j\" (UID: \"3537a2cf-bd5a-41e3-8f8e-01b066e279d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"
Sep 10 23:08:22.511366 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:22.511342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdb9g\" (UniqueName: \"kubernetes.io/projected/3537a2cf-bd5a-41e3-8f8e-01b066e279d3-kube-api-access-cdb9g\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dnr2j\" (UID: \"3537a2cf-bd5a-41e3-8f8e-01b066e279d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"
Sep 10 23:08:22.615146 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:22.615097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"
Sep 10 23:08:22.902427 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:22.902351    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"]
Sep 10 23:08:22.906075 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:08:22.906027    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3537a2cf_bd5a_41e3_8f8e_01b066e279d3.slice/crio-a2ce40f3d498fcfb682b9cc409d951701042e812f80d722aee7bc103d2e80968 WatchSource:0}: Error finding container a2ce40f3d498fcfb682b9cc409d951701042e812f80d722aee7bc103d2e80968: Status 404 returned error can't find the container with id a2ce40f3d498fcfb682b9cc409d951701042e812f80d722aee7bc103d2e80968
Sep 10 23:08:23.772139 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:23.772104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j" event={"ID":"3537a2cf-bd5a-41e3-8f8e-01b066e279d3","Type":"ContainerStarted","Data":"a2ce40f3d498fcfb682b9cc409d951701042e812f80d722aee7bc103d2e80968"}
Sep 10 23:08:27.794238 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:27.794204    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j" event={"ID":"3537a2cf-bd5a-41e3-8f8e-01b066e279d3","Type":"ContainerStarted","Data":"baab1a485344336e641bc173c1477740eb1ddcc2da6914a8e0ed9b75171fefbd"}
Sep 10 23:08:27.794647 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:27.794328    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"
Sep 10 23:08:27.839725 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:27.839665    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j" podStartSLOduration=1.733106545 podStartE2EDuration="5.839635281s" totalImagesPullingTime="4.106528736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:08:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:08:27.837296491 +0000 UTC m=+428.528873696" watchObservedRunningTime="2026-09-10 23:08:27.839635281 +0000 UTC m=+428.531212467"
Sep 10 23:08:29.743848 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:29.743818    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"]
Sep 10 23:08:29.747001 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:29.746981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"
Sep 10 23:08:29.753790 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:29.753766    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 23:08:29.754241 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:29.754224    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6kc25\""
Sep 10 23:08:29.754740 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:29.754724    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 23:08:29.761345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:29.761324    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"]
Sep 10 23:08:29.886831 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:29.886791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/559d95fc-38d9-4e28-a02d-6d577abbefdd-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bd4s9\" (UID: \"559d95fc-38d9-4e28-a02d-6d577abbefdd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"
Sep 10 23:08:29.887003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:29.886838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbm75\" (UniqueName: \"kubernetes.io/projected/559d95fc-38d9-4e28-a02d-6d577abbefdd-kube-api-access-pbm75\") pod \"kuadrant-console-plugin-fbdd4f445-bd4s9\" (UID: \"559d95fc-38d9-4e28-a02d-6d577abbefdd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"
Sep 10 23:08:29.887003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:29.886898    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/559d95fc-38d9-4e28-a02d-6d577abbefdd-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bd4s9\" (UID: \"559d95fc-38d9-4e28-a02d-6d577abbefdd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"
Sep 10 23:08:29.988348 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:29.988304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/559d95fc-38d9-4e28-a02d-6d577abbefdd-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bd4s9\" (UID: \"559d95fc-38d9-4e28-a02d-6d577abbefdd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"
Sep 10 23:08:29.988539 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:29.988364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbm75\" (UniqueName: \"kubernetes.io/projected/559d95fc-38d9-4e28-a02d-6d577abbefdd-kube-api-access-pbm75\") pod \"kuadrant-console-plugin-fbdd4f445-bd4s9\" (UID: \"559d95fc-38d9-4e28-a02d-6d577abbefdd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"
Sep 10 23:08:29.988539 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:29.988409    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/559d95fc-38d9-4e28-a02d-6d577abbefdd-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bd4s9\" (UID: \"559d95fc-38d9-4e28-a02d-6d577abbefdd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"
Sep 10 23:08:29.988662 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:08:29.988602    2563 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 10 23:08:29.988744 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:08:29.988691    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/559d95fc-38d9-4e28-a02d-6d577abbefdd-plugin-serving-cert podName:559d95fc-38d9-4e28-a02d-6d577abbefdd nodeName:}" failed. No retries permitted until 2026-09-10 23:08:30.488667248 +0000 UTC m=+431.180244427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/559d95fc-38d9-4e28-a02d-6d577abbefdd-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-bd4s9" (UID: "559d95fc-38d9-4e28-a02d-6d577abbefdd") : secret "plugin-serving-cert" not found
Sep 10 23:08:29.989028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:29.989008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/559d95fc-38d9-4e28-a02d-6d577abbefdd-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-bd4s9\" (UID: \"559d95fc-38d9-4e28-a02d-6d577abbefdd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"
Sep 10 23:08:29.997789 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:29.997743    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbm75\" (UniqueName: \"kubernetes.io/projected/559d95fc-38d9-4e28-a02d-6d577abbefdd-kube-api-access-pbm75\") pod \"kuadrant-console-plugin-fbdd4f445-bd4s9\" (UID: \"559d95fc-38d9-4e28-a02d-6d577abbefdd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"
Sep 10 23:08:30.492884 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:30.492826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/559d95fc-38d9-4e28-a02d-6d577abbefdd-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bd4s9\" (UID: \"559d95fc-38d9-4e28-a02d-6d577abbefdd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"
Sep 10 23:08:30.495247 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:30.495220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/559d95fc-38d9-4e28-a02d-6d577abbefdd-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-bd4s9\" (UID: \"559d95fc-38d9-4e28-a02d-6d577abbefdd\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"
Sep 10 23:08:30.665423 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:30.665385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"
Sep 10 23:08:30.935953 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:30.935927    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9"]
Sep 10 23:08:30.938930 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:08:30.938903    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod559d95fc_38d9_4e28_a02d_6d577abbefdd.slice/crio-3914ffcbf6885896f59d286f6d8afa59dfe0efb002a09955dba4d2cf9f03356e WatchSource:0}: Error finding container 3914ffcbf6885896f59d286f6d8afa59dfe0efb002a09955dba4d2cf9f03356e: Status 404 returned error can't find the container with id 3914ffcbf6885896f59d286f6d8afa59dfe0efb002a09955dba4d2cf9f03356e
Sep 10 23:08:31.819273 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:31.819214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9" event={"ID":"559d95fc-38d9-4e28-a02d-6d577abbefdd","Type":"ContainerStarted","Data":"3914ffcbf6885896f59d286f6d8afa59dfe0efb002a09955dba4d2cf9f03356e"}
Sep 10 23:08:35.839918 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:35.839881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9" event={"ID":"559d95fc-38d9-4e28-a02d-6d577abbefdd","Type":"ContainerStarted","Data":"b9b8a71253993ba746bb51575a260ad8aa6e19c2eb20004c0c978139faafb71e"}
Sep 10 23:08:35.858277 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:35.858224    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-bd4s9" podStartSLOduration=2.458714359 podStartE2EDuration="6.858210345s" totalImagesPullingTime="4.399495986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:08:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:08:35.858138058 +0000 UTC m=+436.549715248" watchObservedRunningTime="2026-09-10 23:08:35.858210345 +0000 UTC m=+436.549787529"
Sep 10 23:08:38.800939 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:38.800909    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"
Sep 10 23:08:40.895257 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:40.895224    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"]
Sep 10 23:08:40.895706 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:40.895418    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j" podUID="3537a2cf-bd5a-41e3-8f8e-01b066e279d3" containerName="manager" containerID="cri-o://baab1a485344336e641bc173c1477740eb1ddcc2da6914a8e0ed9b75171fefbd" gracePeriod=2
Sep 10 23:08:40.900155 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:40.900127    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"]
Sep 10 23:08:40.980942 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:40.980910    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-lqn4f"]
Sep 10 23:08:40.981327 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:40.981310    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3537a2cf-bd5a-41e3-8f8e-01b066e279d3" containerName="manager"
Sep 10 23:08:40.981409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:40.981330    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3537a2cf-bd5a-41e3-8f8e-01b066e279d3" containerName="manager"
Sep 10 23:08:40.981409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:40.981404    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3537a2cf-bd5a-41e3-8f8e-01b066e279d3" containerName="manager"
Sep 10 23:08:40.984229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:40.984209    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lqn4f"
Sep 10 23:08:40.986586 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:40.986556    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-whd78\""
Sep 10 23:08:40.997465 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:40.997445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-lqn4f"]
Sep 10 23:08:41.021957 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:08:41.021925    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dnr2j\" is forbidden: User \"system:node:ip-10-0-130-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-246.ec2.internal' and this object" podUID="3537a2cf-bd5a-41e3-8f8e-01b066e279d3" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"
Sep 10 23:08:41.094055 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.094020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62klp\" (UniqueName: \"kubernetes.io/projected/4004b71a-b778-433a-a33b-9c089d353e88-kube-api-access-62klp\") pod \"limitador-operator-controller-manager-74495c69f8-lqn4f\" (UID: \"4004b71a-b778-433a-a33b-9c089d353e88\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lqn4f"
Sep 10 23:08:41.132222 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.132197    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"
Sep 10 23:08:41.134687 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:08:41.134662    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-dnr2j\" is forbidden: User \"system:node:ip-10-0-130-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-246.ec2.internal' and this object" podUID="3537a2cf-bd5a-41e3-8f8e-01b066e279d3" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"
Sep 10 23:08:41.194753 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.194695    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3537a2cf-bd5a-41e3-8f8e-01b066e279d3-extensions-socket-volume\") pod \"3537a2cf-bd5a-41e3-8f8e-01b066e279d3\" (UID: \"3537a2cf-bd5a-41e3-8f8e-01b066e279d3\") "
Sep 10 23:08:41.194753 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.194744    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdb9g\" (UniqueName: \"kubernetes.io/projected/3537a2cf-bd5a-41e3-8f8e-01b066e279d3-kube-api-access-cdb9g\") pod \"3537a2cf-bd5a-41e3-8f8e-01b066e279d3\" (UID: \"3537a2cf-bd5a-41e3-8f8e-01b066e279d3\") "
Sep 10 23:08:41.195051 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.195032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62klp\" (UniqueName: \"kubernetes.io/projected/4004b71a-b778-433a-a33b-9c089d353e88-kube-api-access-62klp\") pod \"limitador-operator-controller-manager-74495c69f8-lqn4f\" (UID: \"4004b71a-b778-433a-a33b-9c089d353e88\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lqn4f"
Sep 10 23:08:41.195261 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.195237    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3537a2cf-bd5a-41e3-8f8e-01b066e279d3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3537a2cf-bd5a-41e3-8f8e-01b066e279d3" (UID: "3537a2cf-bd5a-41e3-8f8e-01b066e279d3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:08:41.196839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.196818    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3537a2cf-bd5a-41e3-8f8e-01b066e279d3-kube-api-access-cdb9g" (OuterVolumeSpecName: "kube-api-access-cdb9g") pod "3537a2cf-bd5a-41e3-8f8e-01b066e279d3" (UID: "3537a2cf-bd5a-41e3-8f8e-01b066e279d3"). InnerVolumeSpecName "kube-api-access-cdb9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:08:41.211723 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.211696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62klp\" (UniqueName: \"kubernetes.io/projected/4004b71a-b778-433a-a33b-9c089d353e88-kube-api-access-62klp\") pod \"limitador-operator-controller-manager-74495c69f8-lqn4f\" (UID: \"4004b71a-b778-433a-a33b-9c089d353e88\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lqn4f"
Sep 10 23:08:41.294373 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.294321    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lqn4f"
Sep 10 23:08:41.295685 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.295653    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3537a2cf-bd5a-41e3-8f8e-01b066e279d3-extensions-socket-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:08:41.295764 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.295688    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdb9g\" (UniqueName: \"kubernetes.io/projected/3537a2cf-bd5a-41e3-8f8e-01b066e279d3-kube-api-access-cdb9g\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:08:41.575106 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.575082    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-lqn4f"]
Sep 10 23:08:41.576903 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:08:41.576872    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4004b71a_b778_433a_a33b_9c089d353e88.slice/crio-1cfff2622ac84676c735fb6bfaaf1c1ee8842a8cfce64ac26eafca93df4cacf7 WatchSource:0}: Error finding container 1cfff2622ac84676c735fb6bfaaf1c1ee8842a8cfce64ac26eafca93df4cacf7: Status 404 returned error can't find the container with id 1cfff2622ac84676c735fb6bfaaf1c1ee8842a8cfce64ac26eafca93df4cacf7
Sep 10 23:08:41.813888 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.813857    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3537a2cf-bd5a-41e3-8f8e-01b066e279d3" path="/var/lib/kubelet/pods/3537a2cf-bd5a-41e3-8f8e-01b066e279d3/volumes"
Sep 10 23:08:41.868528 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.868422    2563 generic.go:356] "Generic (PLEG): container finished" podID="3537a2cf-bd5a-41e3-8f8e-01b066e279d3" containerID="baab1a485344336e641bc173c1477740eb1ddcc2da6914a8e0ed9b75171fefbd" exitCode=0
Sep 10 23:08:41.868528 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.868465    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dnr2j"
Sep 10 23:08:41.868819 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.868547    2563 scope.go:117] "RemoveContainer" containerID="baab1a485344336e641bc173c1477740eb1ddcc2da6914a8e0ed9b75171fefbd"
Sep 10 23:08:41.870270 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.870241    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lqn4f" event={"ID":"4004b71a-b778-433a-a33b-9c089d353e88","Type":"ContainerStarted","Data":"1cfff2622ac84676c735fb6bfaaf1c1ee8842a8cfce64ac26eafca93df4cacf7"}
Sep 10 23:08:41.876644 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.876626    2563 scope.go:117] "RemoveContainer" containerID="baab1a485344336e641bc173c1477740eb1ddcc2da6914a8e0ed9b75171fefbd"
Sep 10 23:08:41.876913 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:08:41.876896    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baab1a485344336e641bc173c1477740eb1ddcc2da6914a8e0ed9b75171fefbd\": container with ID starting with baab1a485344336e641bc173c1477740eb1ddcc2da6914a8e0ed9b75171fefbd not found: ID does not exist" containerID="baab1a485344336e641bc173c1477740eb1ddcc2da6914a8e0ed9b75171fefbd"
Sep 10 23:08:41.876953 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:41.876923    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baab1a485344336e641bc173c1477740eb1ddcc2da6914a8e0ed9b75171fefbd"} err="failed to get container status \"baab1a485344336e641bc173c1477740eb1ddcc2da6914a8e0ed9b75171fefbd\": rpc error: code = NotFound desc = could not find container \"baab1a485344336e641bc173c1477740eb1ddcc2da6914a8e0ed9b75171fefbd\": container with ID starting with baab1a485344336e641bc173c1477740eb1ddcc2da6914a8e0ed9b75171fefbd not found: ID does not exist"
Sep 10 23:08:43.882177 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:43.882140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lqn4f" event={"ID":"4004b71a-b778-433a-a33b-9c089d353e88","Type":"ContainerStarted","Data":"c3040e851d1c962854ed329c5c732bf53b83ce42e76b90c20551d1474ecd748c"}
Sep 10 23:08:43.882663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:43.882265    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lqn4f"
Sep 10 23:08:43.901248 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:43.901199    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lqn4f" podStartSLOduration=2.478148687 podStartE2EDuration="3.901185689s" totalImagesPullingTime="1.423037002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:08:43.899624094 +0000 UTC m=+444.591201304" watchObservedRunningTime="2026-09-10 23:08:43.901185689 +0000 UTC m=+444.592762873"
Sep 10 23:08:54.888515 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:08:54.888463    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-lqn4f"
Sep 10 23:09:27.979192 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:27.979159    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-krjjk"]
Sep 10 23:09:27.983790 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:27.983771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-krjjk"
Sep 10 23:09:27.986896 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:27.986867    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7cz9b\""
Sep 10 23:09:27.994987 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:27.994966    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-krjjk"]
Sep 10 23:09:27.998539 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:27.998510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glfbc\" (UniqueName: \"kubernetes.io/projected/def4cfc2-28eb-4132-9df0-aba88d5c7c52-kube-api-access-glfbc\") pod \"authorino-7bcfd8f9cd-krjjk\" (UID: \"def4cfc2-28eb-4132-9df0-aba88d5c7c52\") " pod="kuadrant-system/authorino-7bcfd8f9cd-krjjk"
Sep 10 23:09:28.099641 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:28.099604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glfbc\" (UniqueName: \"kubernetes.io/projected/def4cfc2-28eb-4132-9df0-aba88d5c7c52-kube-api-access-glfbc\") pod \"authorino-7bcfd8f9cd-krjjk\" (UID: \"def4cfc2-28eb-4132-9df0-aba88d5c7c52\") " pod="kuadrant-system/authorino-7bcfd8f9cd-krjjk"
Sep 10 23:09:28.107887 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:28.107860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glfbc\" (UniqueName: \"kubernetes.io/projected/def4cfc2-28eb-4132-9df0-aba88d5c7c52-kube-api-access-glfbc\") pod \"authorino-7bcfd8f9cd-krjjk\" (UID: \"def4cfc2-28eb-4132-9df0-aba88d5c7c52\") " pod="kuadrant-system/authorino-7bcfd8f9cd-krjjk"
Sep 10 23:09:28.156959 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:28.156931    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-nrgpj"]
Sep 10 23:09:28.160154 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:28.160138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nrgpj"
Sep 10 23:09:28.168055 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:28.168036    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nrgpj"]
Sep 10 23:09:28.200751 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:28.200729    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8q7p\" (UniqueName: \"kubernetes.io/projected/5a12c5ba-d458-489c-b796-71ac1b7e6c6c-kube-api-access-d8q7p\") pod \"authorino-59c974bd48-nrgpj\" (UID: \"5a12c5ba-d458-489c-b796-71ac1b7e6c6c\") " pod="kuadrant-system/authorino-59c974bd48-nrgpj"
Sep 10 23:09:28.293209 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:28.293151    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-krjjk"
Sep 10 23:09:28.301285 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:28.301265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8q7p\" (UniqueName: \"kubernetes.io/projected/5a12c5ba-d458-489c-b796-71ac1b7e6c6c-kube-api-access-d8q7p\") pod \"authorino-59c974bd48-nrgpj\" (UID: \"5a12c5ba-d458-489c-b796-71ac1b7e6c6c\") " pod="kuadrant-system/authorino-59c974bd48-nrgpj"
Sep 10 23:09:28.311325 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:28.311303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8q7p\" (UniqueName: \"kubernetes.io/projected/5a12c5ba-d458-489c-b796-71ac1b7e6c6c-kube-api-access-d8q7p\") pod \"authorino-59c974bd48-nrgpj\" (UID: \"5a12c5ba-d458-489c-b796-71ac1b7e6c6c\") " pod="kuadrant-system/authorino-59c974bd48-nrgpj"
Sep 10 23:09:28.469091 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:28.469062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nrgpj"
Sep 10 23:09:28.569987 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:28.569964    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-krjjk"]
Sep 10 23:09:28.571894 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:09:28.571854    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddef4cfc2_28eb_4132_9df0_aba88d5c7c52.slice/crio-3a8047321758a966f5523a969ab7d5bbadbeb96f097707f896969844f3e17706 WatchSource:0}: Error finding container 3a8047321758a966f5523a969ab7d5bbadbeb96f097707f896969844f3e17706: Status 404 returned error can't find the container with id 3a8047321758a966f5523a969ab7d5bbadbeb96f097707f896969844f3e17706
Sep 10 23:09:28.735882 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:28.735855    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nrgpj"]
Sep 10 23:09:28.739439 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:09:28.739414    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a12c5ba_d458_489c_b796_71ac1b7e6c6c.slice/crio-a9782c502e2c1c9b8b51317de74bd4f38bb3a7b0565a4af026d1e62b0d0408be WatchSource:0}: Error finding container a9782c502e2c1c9b8b51317de74bd4f38bb3a7b0565a4af026d1e62b0d0408be: Status 404 returned error can't find the container with id a9782c502e2c1c9b8b51317de74bd4f38bb3a7b0565a4af026d1e62b0d0408be
Sep 10 23:09:29.082465 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:29.082439    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nrgpj" event={"ID":"5a12c5ba-d458-489c-b796-71ac1b7e6c6c","Type":"ContainerStarted","Data":"a9782c502e2c1c9b8b51317de74bd4f38bb3a7b0565a4af026d1e62b0d0408be"}
Sep 10 23:09:29.083546 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:29.083514    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-krjjk" event={"ID":"def4cfc2-28eb-4132-9df0-aba88d5c7c52","Type":"ContainerStarted","Data":"3a8047321758a966f5523a969ab7d5bbadbeb96f097707f896969844f3e17706"}
Sep 10 23:09:31.094619 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:31.094578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nrgpj" event={"ID":"5a12c5ba-d458-489c-b796-71ac1b7e6c6c","Type":"ContainerStarted","Data":"de8662601f25d5b4ac32d0ab1629e4b9d8dd97db4e78abd568e3743d00947fdc"}
Sep 10 23:09:31.096120 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:31.096089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-krjjk" event={"ID":"def4cfc2-28eb-4132-9df0-aba88d5c7c52","Type":"ContainerStarted","Data":"3d5bf42d6f0738f61a41b37fefc4ae3a0824b46a790fe709671f6dc4ac78645c"}
Sep 10 23:09:31.112124 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:31.112079    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-nrgpj" podStartSLOduration=1.386547861 podStartE2EDuration="3.112065458s" totalImagesPullingTime="1.725517597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:09:31.11065636 +0000 UTC m=+491.802233551" watchObservedRunningTime="2026-09-10 23:09:31.112065458 +0000 UTC m=+491.803642689"
Sep 10 23:09:31.131220 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:31.131167    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-krjjk" podStartSLOduration=2.233583112 podStartE2EDuration="4.131147835s" totalImagesPullingTime="1.897564723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:09:31.130787692 +0000 UTC m=+491.822364875" watchObservedRunningTime="2026-09-10 23:09:31.131147835 +0000 UTC m=+491.822725020"
Sep 10 23:09:31.166673 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:31.166645    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-krjjk"]
Sep 10 23:09:33.105167 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:33.105124    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-krjjk" podUID="def4cfc2-28eb-4132-9df0-aba88d5c7c52" containerName="authorino" containerID="cri-o://3d5bf42d6f0738f61a41b37fefc4ae3a0824b46a790fe709671f6dc4ac78645c" gracePeriod=30
Sep 10 23:09:33.339675 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:33.339653    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-krjjk"
Sep 10 23:09:33.449437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:33.449378    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glfbc\" (UniqueName: \"kubernetes.io/projected/def4cfc2-28eb-4132-9df0-aba88d5c7c52-kube-api-access-glfbc\") pod \"def4cfc2-28eb-4132-9df0-aba88d5c7c52\" (UID: \"def4cfc2-28eb-4132-9df0-aba88d5c7c52\") "
Sep 10 23:09:33.451364 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:33.451339    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def4cfc2-28eb-4132-9df0-aba88d5c7c52-kube-api-access-glfbc" (OuterVolumeSpecName: "kube-api-access-glfbc") pod "def4cfc2-28eb-4132-9df0-aba88d5c7c52" (UID: "def4cfc2-28eb-4132-9df0-aba88d5c7c52"). InnerVolumeSpecName "kube-api-access-glfbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:09:33.550792 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:33.550759    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glfbc\" (UniqueName: \"kubernetes.io/projected/def4cfc2-28eb-4132-9df0-aba88d5c7c52-kube-api-access-glfbc\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:09:34.110876 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:34.110796    2563 generic.go:356] "Generic (PLEG): container finished" podID="def4cfc2-28eb-4132-9df0-aba88d5c7c52" containerID="3d5bf42d6f0738f61a41b37fefc4ae3a0824b46a790fe709671f6dc4ac78645c" exitCode=0
Sep 10 23:09:34.110876 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:34.110834    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-krjjk"
Sep 10 23:09:34.111346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:34.110877    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-krjjk" event={"ID":"def4cfc2-28eb-4132-9df0-aba88d5c7c52","Type":"ContainerDied","Data":"3d5bf42d6f0738f61a41b37fefc4ae3a0824b46a790fe709671f6dc4ac78645c"}
Sep 10 23:09:34.111346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:34.110911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-krjjk" event={"ID":"def4cfc2-28eb-4132-9df0-aba88d5c7c52","Type":"ContainerDied","Data":"3a8047321758a966f5523a969ab7d5bbadbeb96f097707f896969844f3e17706"}
Sep 10 23:09:34.111346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:34.110928    2563 scope.go:117] "RemoveContainer" containerID="3d5bf42d6f0738f61a41b37fefc4ae3a0824b46a790fe709671f6dc4ac78645c"
Sep 10 23:09:34.119191 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:34.119172    2563 scope.go:117] "RemoveContainer" containerID="3d5bf42d6f0738f61a41b37fefc4ae3a0824b46a790fe709671f6dc4ac78645c"
Sep 10 23:09:34.119454 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:09:34.119435    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5bf42d6f0738f61a41b37fefc4ae3a0824b46a790fe709671f6dc4ac78645c\": container with ID starting with 3d5bf42d6f0738f61a41b37fefc4ae3a0824b46a790fe709671f6dc4ac78645c not found: ID does not exist" containerID="3d5bf42d6f0738f61a41b37fefc4ae3a0824b46a790fe709671f6dc4ac78645c"
Sep 10 23:09:34.119543 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:34.119465    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5bf42d6f0738f61a41b37fefc4ae3a0824b46a790fe709671f6dc4ac78645c"} err="failed to get container status \"3d5bf42d6f0738f61a41b37fefc4ae3a0824b46a790fe709671f6dc4ac78645c\": rpc error: code = NotFound desc = could not find container \"3d5bf42d6f0738f61a41b37fefc4ae3a0824b46a790fe709671f6dc4ac78645c\": container with ID starting with 3d5bf42d6f0738f61a41b37fefc4ae3a0824b46a790fe709671f6dc4ac78645c not found: ID does not exist"
Sep 10 23:09:34.127759 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:34.127739    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-krjjk"]
Sep 10 23:09:34.132965 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:34.132946    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-krjjk"]
Sep 10 23:09:35.812993 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:35.812962    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="def4cfc2-28eb-4132-9df0-aba88d5c7c52" path="/var/lib/kubelet/pods/def4cfc2-28eb-4132-9df0-aba88d5c7c52/volumes"
Sep 10 23:09:56.025634 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.025589    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5hf8q"]
Sep 10 23:09:56.026112 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.025945    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def4cfc2-28eb-4132-9df0-aba88d5c7c52" containerName="authorino"
Sep 10 23:09:56.026112 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.025957    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="def4cfc2-28eb-4132-9df0-aba88d5c7c52" containerName="authorino"
Sep 10 23:09:56.026112 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.026029    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="def4cfc2-28eb-4132-9df0-aba88d5c7c52" containerName="authorino"
Sep 10 23:09:56.029188 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.029170    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-5hf8q"
Sep 10 23:09:56.031616 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.031586    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 23:09:56.037851 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.037832    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5hf8q"]
Sep 10 23:09:56.139151 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.139120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef158e43-d46d-4eba-9554-ba855a54e889-tls-cert\") pod \"authorino-6d4dfc649f-5hf8q\" (UID: \"ef158e43-d46d-4eba-9554-ba855a54e889\") " pod="kuadrant-system/authorino-6d4dfc649f-5hf8q"
Sep 10 23:09:56.139300 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.139215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5gwb\" (UniqueName: \"kubernetes.io/projected/ef158e43-d46d-4eba-9554-ba855a54e889-kube-api-access-q5gwb\") pod \"authorino-6d4dfc649f-5hf8q\" (UID: \"ef158e43-d46d-4eba-9554-ba855a54e889\") " pod="kuadrant-system/authorino-6d4dfc649f-5hf8q"
Sep 10 23:09:56.141506 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.141463    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5hf8q"]
Sep 10 23:09:56.141730 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:09:56.141706    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-q5gwb tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-5hf8q" podUID="ef158e43-d46d-4eba-9554-ba855a54e889"
Sep 10 23:09:56.211090 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.211044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-5hf8q"
Sep 10 23:09:56.217922 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.217903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-5hf8q"
Sep 10 23:09:56.240292 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.240265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gwb\" (UniqueName: \"kubernetes.io/projected/ef158e43-d46d-4eba-9554-ba855a54e889-kube-api-access-q5gwb\") pod \"authorino-6d4dfc649f-5hf8q\" (UID: \"ef158e43-d46d-4eba-9554-ba855a54e889\") " pod="kuadrant-system/authorino-6d4dfc649f-5hf8q"
Sep 10 23:09:56.240384 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.240318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef158e43-d46d-4eba-9554-ba855a54e889-tls-cert\") pod \"authorino-6d4dfc649f-5hf8q\" (UID: \"ef158e43-d46d-4eba-9554-ba855a54e889\") " pod="kuadrant-system/authorino-6d4dfc649f-5hf8q"
Sep 10 23:09:56.242723 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.242695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef158e43-d46d-4eba-9554-ba855a54e889-tls-cert\") pod \"authorino-6d4dfc649f-5hf8q\" (UID: \"ef158e43-d46d-4eba-9554-ba855a54e889\") " pod="kuadrant-system/authorino-6d4dfc649f-5hf8q"
Sep 10 23:09:56.248053 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.248029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gwb\" (UniqueName: \"kubernetes.io/projected/ef158e43-d46d-4eba-9554-ba855a54e889-kube-api-access-q5gwb\") pod \"authorino-6d4dfc649f-5hf8q\" (UID: \"ef158e43-d46d-4eba-9554-ba855a54e889\") " pod="kuadrant-system/authorino-6d4dfc649f-5hf8q"
Sep 10 23:09:56.341213 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.341188    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5gwb\" (UniqueName: \"kubernetes.io/projected/ef158e43-d46d-4eba-9554-ba855a54e889-kube-api-access-q5gwb\") pod \"ef158e43-d46d-4eba-9554-ba855a54e889\" (UID: \"ef158e43-d46d-4eba-9554-ba855a54e889\") "
Sep 10 23:09:56.341327 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.341249    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef158e43-d46d-4eba-9554-ba855a54e889-tls-cert\") pod \"ef158e43-d46d-4eba-9554-ba855a54e889\" (UID: \"ef158e43-d46d-4eba-9554-ba855a54e889\") "
Sep 10 23:09:56.343074 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.343047    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef158e43-d46d-4eba-9554-ba855a54e889-kube-api-access-q5gwb" (OuterVolumeSpecName: "kube-api-access-q5gwb") pod "ef158e43-d46d-4eba-9554-ba855a54e889" (UID: "ef158e43-d46d-4eba-9554-ba855a54e889"). InnerVolumeSpecName "kube-api-access-q5gwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:09:56.343156 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.343116    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef158e43-d46d-4eba-9554-ba855a54e889-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ef158e43-d46d-4eba-9554-ba855a54e889" (UID: "ef158e43-d46d-4eba-9554-ba855a54e889"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:09:56.442360 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.442333    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5gwb\" (UniqueName: \"kubernetes.io/projected/ef158e43-d46d-4eba-9554-ba855a54e889-kube-api-access-q5gwb\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:09:56.442360 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:56.442356    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef158e43-d46d-4eba-9554-ba855a54e889-tls-cert\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:09:57.215175 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:57.215149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-5hf8q"
Sep 10 23:09:57.246227 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:57.246190    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5hf8q"]
Sep 10 23:09:57.248550 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:57.248526    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-5hf8q"]
Sep 10 23:09:57.813729 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:57.813700    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef158e43-d46d-4eba-9554-ba855a54e889" path="/var/lib/kubelet/pods/ef158e43-d46d-4eba-9554-ba855a54e889/volumes"
Sep 10 23:09:59.402217 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:59.402181    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nrgpj"]
Sep 10 23:09:59.402618 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:59.402360    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-nrgpj" podUID="5a12c5ba-d458-489c-b796-71ac1b7e6c6c" containerName="authorino" containerID="cri-o://de8662601f25d5b4ac32d0ab1629e4b9d8dd97db4e78abd568e3743d00947fdc" gracePeriod=30
Sep 10 23:09:59.648534 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:59.648511    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nrgpj"
Sep 10 23:09:59.769278 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:59.769192    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8q7p\" (UniqueName: \"kubernetes.io/projected/5a12c5ba-d458-489c-b796-71ac1b7e6c6c-kube-api-access-d8q7p\") pod \"5a12c5ba-d458-489c-b796-71ac1b7e6c6c\" (UID: \"5a12c5ba-d458-489c-b796-71ac1b7e6c6c\") "
Sep 10 23:09:59.771277 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:59.771252    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a12c5ba-d458-489c-b796-71ac1b7e6c6c-kube-api-access-d8q7p" (OuterVolumeSpecName: "kube-api-access-d8q7p") pod "5a12c5ba-d458-489c-b796-71ac1b7e6c6c" (UID: "5a12c5ba-d458-489c-b796-71ac1b7e6c6c"). InnerVolumeSpecName "kube-api-access-d8q7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:09:59.870424 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:09:59.870392    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8q7p\" (UniqueName: \"kubernetes.io/projected/5a12c5ba-d458-489c-b796-71ac1b7e6c6c-kube-api-access-d8q7p\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:10:00.229783 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:00.229747    2563 generic.go:356] "Generic (PLEG): container finished" podID="5a12c5ba-d458-489c-b796-71ac1b7e6c6c" containerID="de8662601f25d5b4ac32d0ab1629e4b9d8dd97db4e78abd568e3743d00947fdc" exitCode=0
Sep 10 23:10:00.229984 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:00.229789    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-nrgpj"
Sep 10 23:10:00.229984 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:00.229840    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nrgpj" event={"ID":"5a12c5ba-d458-489c-b796-71ac1b7e6c6c","Type":"ContainerDied","Data":"de8662601f25d5b4ac32d0ab1629e4b9d8dd97db4e78abd568e3743d00947fdc"}
Sep 10 23:10:00.229984 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:00.229884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-nrgpj" event={"ID":"5a12c5ba-d458-489c-b796-71ac1b7e6c6c","Type":"ContainerDied","Data":"a9782c502e2c1c9b8b51317de74bd4f38bb3a7b0565a4af026d1e62b0d0408be"}
Sep 10 23:10:00.229984 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:00.229905    2563 scope.go:117] "RemoveContainer" containerID="de8662601f25d5b4ac32d0ab1629e4b9d8dd97db4e78abd568e3743d00947fdc"
Sep 10 23:10:00.238121 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:00.238103    2563 scope.go:117] "RemoveContainer" containerID="de8662601f25d5b4ac32d0ab1629e4b9d8dd97db4e78abd568e3743d00947fdc"
Sep 10 23:10:00.238397 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:10:00.238378    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8662601f25d5b4ac32d0ab1629e4b9d8dd97db4e78abd568e3743d00947fdc\": container with ID starting with de8662601f25d5b4ac32d0ab1629e4b9d8dd97db4e78abd568e3743d00947fdc not found: ID does not exist" containerID="de8662601f25d5b4ac32d0ab1629e4b9d8dd97db4e78abd568e3743d00947fdc"
Sep 10 23:10:00.238437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:00.238405    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8662601f25d5b4ac32d0ab1629e4b9d8dd97db4e78abd568e3743d00947fdc"} err="failed to get container status \"de8662601f25d5b4ac32d0ab1629e4b9d8dd97db4e78abd568e3743d00947fdc\": rpc error: code = NotFound desc = could not find container \"de8662601f25d5b4ac32d0ab1629e4b9d8dd97db4e78abd568e3743d00947fdc\": container with ID starting with de8662601f25d5b4ac32d0ab1629e4b9d8dd97db4e78abd568e3743d00947fdc not found: ID does not exist"
Sep 10 23:10:00.248029 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:00.248009    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nrgpj"]
Sep 10 23:10:00.252240 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:00.252218    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-nrgpj"]
Sep 10 23:10:01.813694 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:01.813654    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a12c5ba-d458-489c-b796-71ac1b7e6c6c" path="/var/lib/kubelet/pods/5a12c5ba-d458-489c-b796-71ac1b7e6c6c/volumes"
Sep 10 23:10:41.910951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:41.910913    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mcn47"]
Sep 10 23:10:41.911608 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:41.911263    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a12c5ba-d458-489c-b796-71ac1b7e6c6c" containerName="authorino"
Sep 10 23:10:41.911608 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:41.911275    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a12c5ba-d458-489c-b796-71ac1b7e6c6c" containerName="authorino"
Sep 10 23:10:41.911608 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:41.911334    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a12c5ba-d458-489c-b796-71ac1b7e6c6c" containerName="authorino"
Sep 10 23:10:41.915659 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:41.915638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-mcn47"
Sep 10 23:10:41.918476 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:41.918454    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 23:10:41.918668 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:41.918649    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hzhzk\""
Sep 10 23:10:41.926914 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:41.926891    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mcn47"]
Sep 10 23:10:42.037164 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:42.037127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2dc4813-36ba-4324-bf13-4e83f11ded1d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mcn47\" (UID: \"a2dc4813-36ba-4324-bf13-4e83f11ded1d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mcn47"
Sep 10 23:10:42.037527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:42.037474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg98m\" (UniqueName: \"kubernetes.io/projected/a2dc4813-36ba-4324-bf13-4e83f11ded1d-kube-api-access-sg98m\") pod \"payload-processing-5b54b7fbb7-mcn47\" (UID: \"a2dc4813-36ba-4324-bf13-4e83f11ded1d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mcn47"
Sep 10 23:10:42.138617 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:42.138586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2dc4813-36ba-4324-bf13-4e83f11ded1d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mcn47\" (UID: \"a2dc4813-36ba-4324-bf13-4e83f11ded1d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mcn47"
Sep 10 23:10:42.138783 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:42.138650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg98m\" (UniqueName: \"kubernetes.io/projected/a2dc4813-36ba-4324-bf13-4e83f11ded1d-kube-api-access-sg98m\") pod \"payload-processing-5b54b7fbb7-mcn47\" (UID: \"a2dc4813-36ba-4324-bf13-4e83f11ded1d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mcn47"
Sep 10 23:10:42.139281 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:42.139262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2dc4813-36ba-4324-bf13-4e83f11ded1d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mcn47\" (UID: \"a2dc4813-36ba-4324-bf13-4e83f11ded1d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mcn47"
Sep 10 23:10:42.149897 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:42.149870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg98m\" (UniqueName: \"kubernetes.io/projected/a2dc4813-36ba-4324-bf13-4e83f11ded1d-kube-api-access-sg98m\") pod \"payload-processing-5b54b7fbb7-mcn47\" (UID: \"a2dc4813-36ba-4324-bf13-4e83f11ded1d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mcn47"
Sep 10 23:10:42.225792 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:42.225717    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-mcn47"
Sep 10 23:10:42.496967 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:42.496944    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mcn47"]
Sep 10 23:10:42.499007 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:10:42.498982    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dc4813_36ba_4324_bf13_4e83f11ded1d.slice/crio-2f377859df76147394ab60c88911861c3bcc9a264fd10da2bad4a902e772566f WatchSource:0}: Error finding container 2f377859df76147394ab60c88911861c3bcc9a264fd10da2bad4a902e772566f: Status 404 returned error can't find the container with id 2f377859df76147394ab60c88911861c3bcc9a264fd10da2bad4a902e772566f
Sep 10 23:10:43.425445 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:43.425406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-mcn47" event={"ID":"a2dc4813-36ba-4324-bf13-4e83f11ded1d","Type":"ContainerStarted","Data":"2f377859df76147394ab60c88911861c3bcc9a264fd10da2bad4a902e772566f"}
Sep 10 23:10:45.437201 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:45.437162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-mcn47" event={"ID":"a2dc4813-36ba-4324-bf13-4e83f11ded1d","Type":"ContainerStarted","Data":"de5d4dfc5801c036236c1030872c9d9a7cd97c8774118979b21da361eaf9c8ec"}
Sep 10 23:10:45.437689 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:45.437374    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-mcn47"
Sep 10 23:10:45.455525 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:45.455472    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-mcn47" podStartSLOduration=2.330378975 podStartE2EDuration="4.455454586s" totalImagesPullingTime="2.125075611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:10:45.453998159 +0000 UTC m=+566.145575365" watchObservedRunningTime="2026-09-10 23:10:45.455454586 +0000 UTC m=+566.147031774"
Sep 10 23:10:56.443453 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:10:56.443422    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-mcn47"
Sep 10 23:11:17.166529 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.166476    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"]
Sep 10 23:11:17.169866 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.169847    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.173169 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.173141    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 10 23:11:17.173321 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.173250    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 23:11:17.173321 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.173260    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dnpmw\""
Sep 10 23:11:17.173476 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.173356    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 23:11:17.186094 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.186066    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"]
Sep 10 23:11:17.260960 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.260925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pnr4\" (UniqueName: \"kubernetes.io/projected/d38e4409-bcc9-4c8f-97d5-072a43d512c8-kube-api-access-6pnr4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.261146 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.260993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d38e4409-bcc9-4c8f-97d5-072a43d512c8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.261146 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.261048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d38e4409-bcc9-4c8f-97d5-072a43d512c8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.261146 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.261086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d38e4409-bcc9-4c8f-97d5-072a43d512c8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.261146 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.261110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d38e4409-bcc9-4c8f-97d5-072a43d512c8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.261146 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.261129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d38e4409-bcc9-4c8f-97d5-072a43d512c8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.361703 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.361663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pnr4\" (UniqueName: \"kubernetes.io/projected/d38e4409-bcc9-4c8f-97d5-072a43d512c8-kube-api-access-6pnr4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.361903 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.361740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d38e4409-bcc9-4c8f-97d5-072a43d512c8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.361903 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.361758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d38e4409-bcc9-4c8f-97d5-072a43d512c8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.361903 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.361777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d38e4409-bcc9-4c8f-97d5-072a43d512c8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.361903 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.361795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d38e4409-bcc9-4c8f-97d5-072a43d512c8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.361903 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.361811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d38e4409-bcc9-4c8f-97d5-072a43d512c8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.362185 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.362165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d38e4409-bcc9-4c8f-97d5-072a43d512c8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.362222 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.362205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d38e4409-bcc9-4c8f-97d5-072a43d512c8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.362255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.362239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d38e4409-bcc9-4c8f-97d5-072a43d512c8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.364075 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.364054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d38e4409-bcc9-4c8f-97d5-072a43d512c8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.364351 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.364333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d38e4409-bcc9-4c8f-97d5-072a43d512c8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.382191 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.382159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pnr4\" (UniqueName: \"kubernetes.io/projected/d38e4409-bcc9-4c8f-97d5-072a43d512c8-kube-api-access-6pnr4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc\" (UID: \"d38e4409-bcc9-4c8f-97d5-072a43d512c8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.480465 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.480375    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:17.755512 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:17.755400    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"]
Sep 10 23:11:17.758340 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:11:17.758305    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38e4409_bcc9_4c8f_97d5_072a43d512c8.slice/crio-06e38a7ce85afaebf6dfcc41b710b0284d1c97b65c0b79d240beffbb047aab26 WatchSource:0}: Error finding container 06e38a7ce85afaebf6dfcc41b710b0284d1c97b65c0b79d240beffbb047aab26: Status 404 returned error can't find the container with id 06e38a7ce85afaebf6dfcc41b710b0284d1c97b65c0b79d240beffbb047aab26
Sep 10 23:11:18.582314 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:18.582264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc" event={"ID":"d38e4409-bcc9-4c8f-97d5-072a43d512c8","Type":"ContainerStarted","Data":"06e38a7ce85afaebf6dfcc41b710b0284d1c97b65c0b79d240beffbb047aab26"}
Sep 10 23:11:19.588733 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:19.588692    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc" event={"ID":"d38e4409-bcc9-4c8f-97d5-072a43d512c8","Type":"ContainerStarted","Data":"40a91fe89c38235d815692d03de91e19ee17512061a0b278f74aa7c50724b6b1"}
Sep 10 23:11:19.609781 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:19.609735    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc" podStartSLOduration=1.485846512 podStartE2EDuration="2.609719926s" totalImagesPullingTime="1.123873414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:11:19.60834784 +0000 UTC m=+600.299925058" watchObservedRunningTime="2026-09-10 23:11:19.609719926 +0000 UTC m=+600.301297110"
Sep 10 23:11:19.771484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:19.771458    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:11:19.771781 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:19.771764    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:11:27.481249 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:27.481216    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:27.483855 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:27.481279    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:27.493923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:27.493900    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:11:27.637423 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:11:27.637389    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc"
Sep 10 23:13:48.328957 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:48.328869    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7555cfcc9b-7rpzq"]
Sep 10 23:13:48.332230 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:48.332212    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7555cfcc9b-7rpzq"
Sep 10 23:13:48.335390 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:48.335371    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 23:13:48.335523 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:48.335444    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7cz9b\""
Sep 10 23:13:48.340734 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:48.340709    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7555cfcc9b-7rpzq"]
Sep 10 23:13:48.504822 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:48.504783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxrq8\" (UniqueName: \"kubernetes.io/projected/be49a481-46c0-41f9-b5f3-ee9f4e4e91ad-kube-api-access-jxrq8\") pod \"authorino-7555cfcc9b-7rpzq\" (UID: \"be49a481-46c0-41f9-b5f3-ee9f4e4e91ad\") " pod="kuadrant-system/authorino-7555cfcc9b-7rpzq"
Sep 10 23:13:48.505012 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:48.504863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/be49a481-46c0-41f9-b5f3-ee9f4e4e91ad-tls-cert\") pod \"authorino-7555cfcc9b-7rpzq\" (UID: \"be49a481-46c0-41f9-b5f3-ee9f4e4e91ad\") " pod="kuadrant-system/authorino-7555cfcc9b-7rpzq"
Sep 10 23:13:48.606182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:48.606086    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/be49a481-46c0-41f9-b5f3-ee9f4e4e91ad-tls-cert\") pod \"authorino-7555cfcc9b-7rpzq\" (UID: \"be49a481-46c0-41f9-b5f3-ee9f4e4e91ad\") " pod="kuadrant-system/authorino-7555cfcc9b-7rpzq"
Sep 10 23:13:48.606182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:48.606160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxrq8\" (UniqueName: \"kubernetes.io/projected/be49a481-46c0-41f9-b5f3-ee9f4e4e91ad-kube-api-access-jxrq8\") pod \"authorino-7555cfcc9b-7rpzq\" (UID: \"be49a481-46c0-41f9-b5f3-ee9f4e4e91ad\") " pod="kuadrant-system/authorino-7555cfcc9b-7rpzq"
Sep 10 23:13:48.608565 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:48.608542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/be49a481-46c0-41f9-b5f3-ee9f4e4e91ad-tls-cert\") pod \"authorino-7555cfcc9b-7rpzq\" (UID: \"be49a481-46c0-41f9-b5f3-ee9f4e4e91ad\") " pod="kuadrant-system/authorino-7555cfcc9b-7rpzq"
Sep 10 23:13:48.614507 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:48.614461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxrq8\" (UniqueName: \"kubernetes.io/projected/be49a481-46c0-41f9-b5f3-ee9f4e4e91ad-kube-api-access-jxrq8\") pod \"authorino-7555cfcc9b-7rpzq\" (UID: \"be49a481-46c0-41f9-b5f3-ee9f4e4e91ad\") " pod="kuadrant-system/authorino-7555cfcc9b-7rpzq"
Sep 10 23:13:48.642006 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:48.641977    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7555cfcc9b-7rpzq"
Sep 10 23:13:48.910892 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:48.910837    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7555cfcc9b-7rpzq"]
Sep 10 23:13:48.913561 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:13:48.913534    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe49a481_46c0_41f9_b5f3_ee9f4e4e91ad.slice/crio-1e1fae1c95ab890a5b1ac9ec6782173a4a3864f2b55b2e231aabc31f042a8577 WatchSource:0}: Error finding container 1e1fae1c95ab890a5b1ac9ec6782173a4a3864f2b55b2e231aabc31f042a8577: Status 404 returned error can't find the container with id 1e1fae1c95ab890a5b1ac9ec6782173a4a3864f2b55b2e231aabc31f042a8577
Sep 10 23:13:48.914848 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:48.914829    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:13:49.270349 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:49.270264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7555cfcc9b-7rpzq" event={"ID":"be49a481-46c0-41f9-b5f3-ee9f4e4e91ad","Type":"ContainerStarted","Data":"1e1fae1c95ab890a5b1ac9ec6782173a4a3864f2b55b2e231aabc31f042a8577"}
Sep 10 23:13:50.276768 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:50.276680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7555cfcc9b-7rpzq" event={"ID":"be49a481-46c0-41f9-b5f3-ee9f4e4e91ad","Type":"ContainerStarted","Data":"88b8751edddf0d5681ecab845ff3a057a5d8e9727d49edc2e9bb7a0234dbad9b"}
Sep 10 23:13:50.294312 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:13:50.294245    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7555cfcc9b-7rpzq" podStartSLOduration=1.196620969 podStartE2EDuration="2.294224222s" totalImagesPullingTime="1.097603253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:13:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:13:50.293438129 +0000 UTC m=+750.985015339" watchObservedRunningTime="2026-09-10 23:13:50.294224222 +0000 UTC m=+750.985801494"
Sep 10 23:14:08.856425 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:08.856387    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7855-a41ca6"]
Sep 10 23:14:08.859959 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:08.859941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7855-a41ca6"
Sep 10 23:14:08.862402 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:08.862378    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f4p5h\""
Sep 10 23:14:08.867058 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:08.866779    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7855-a41ca6"]
Sep 10 23:14:08.875315 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:08.875285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz28d\" (UniqueName: \"kubernetes.io/projected/88ca9996-398f-4845-9b4b-905ca6665f35-kube-api-access-lz28d\") pod \"test-curl-7855-a41ca6\" (UID: \"88ca9996-398f-4845-9b4b-905ca6665f35\") " pod="opendatahub/test-curl-7855-a41ca6"
Sep 10 23:14:08.976760 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:08.976716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz28d\" (UniqueName: \"kubernetes.io/projected/88ca9996-398f-4845-9b4b-905ca6665f35-kube-api-access-lz28d\") pod \"test-curl-7855-a41ca6\" (UID: \"88ca9996-398f-4845-9b4b-905ca6665f35\") " pod="opendatahub/test-curl-7855-a41ca6"
Sep 10 23:14:08.987611 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:08.987579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz28d\" (UniqueName: \"kubernetes.io/projected/88ca9996-398f-4845-9b4b-905ca6665f35-kube-api-access-lz28d\") pod \"test-curl-7855-a41ca6\" (UID: \"88ca9996-398f-4845-9b4b-905ca6665f35\") " pod="opendatahub/test-curl-7855-a41ca6"
Sep 10 23:14:09.170045 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:09.169944    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7855-a41ca6"
Sep 10 23:14:09.440071 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:09.440045    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7855-a41ca6"]
Sep 10 23:14:09.441975 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:14:09.441945    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88ca9996_398f_4845_9b4b_905ca6665f35.slice/crio-0e5b5c094b059d1475551b673dbe7e4604070698885773d84d753fc7665e0aff WatchSource:0}: Error finding container 0e5b5c094b059d1475551b673dbe7e4604070698885773d84d753fc7665e0aff: Status 404 returned error can't find the container with id 0e5b5c094b059d1475551b673dbe7e4604070698885773d84d753fc7665e0aff
Sep 10 23:14:10.372870 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:10.372829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7855-a41ca6" event={"ID":"88ca9996-398f-4845-9b4b-905ca6665f35","Type":"ContainerStarted","Data":"0e5b5c094b059d1475551b673dbe7e4604070698885773d84d753fc7665e0aff"}
Sep 10 23:14:12.383237 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.383197    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7855-a41ca6" event={"ID":"88ca9996-398f-4845-9b4b-905ca6665f35","Type":"ContainerStarted","Data":"856faded591bdeb2f79224054f5aa8470b1b0aa79bf8d367bef8562f2120268f"}
Sep 10 23:14:12.398850 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.398804    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7855-a41ca6" podStartSLOduration=2.145952118 podStartE2EDuration="4.398788822s" totalImagesPullingTime="2.252836704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:12.397969468 +0000 UTC m=+773.089546655" watchObservedRunningTime="2026-09-10 23:14:12.398788822 +0000 UTC m=+773.090366005"
Sep 10 23:14:12.594417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.594378    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"]
Sep 10 23:14:12.597914 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.597893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"
Sep 10 23:14:12.600199 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.600177    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-7025b9\""
Sep 10 23:14:12.608311 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.608288    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"]
Sep 10 23:14:12.634397 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.634323    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"]
Sep 10 23:14:12.637725 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.637705    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"
Sep 10 23:14:12.640117 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.640096    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-7025b9-dockercfg-7gh7s\""
Sep 10 23:14:12.651774 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.651753    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"]
Sep 10 23:14:12.713907 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.713874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg4ln\" (UniqueName: \"kubernetes.io/projected/f4084d5e-45e6-4acf-93d5-a9a9f2150b67-kube-api-access-tg4ln\") pod \"payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7\" (UID: \"f4084d5e-45e6-4acf-93d5-a9a9f2150b67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"
Sep 10 23:14:12.714089 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.713959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5541bf40-af13-42a3-89d3-0c79e2a2b35b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv\" (UID: \"5541bf40-af13-42a3-89d3-0c79e2a2b35b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"
Sep 10 23:14:12.714089 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.713986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q62dg\" (UniqueName: \"kubernetes.io/projected/5541bf40-af13-42a3-89d3-0c79e2a2b35b-kube-api-access-q62dg\") pod \"payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv\" (UID: \"5541bf40-af13-42a3-89d3-0c79e2a2b35b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"
Sep 10 23:14:12.714089 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.714017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4084d5e-45e6-4acf-93d5-a9a9f2150b67-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7\" (UID: \"f4084d5e-45e6-4acf-93d5-a9a9f2150b67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"
Sep 10 23:14:12.814856 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.814819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg4ln\" (UniqueName: \"kubernetes.io/projected/f4084d5e-45e6-4acf-93d5-a9a9f2150b67-kube-api-access-tg4ln\") pod \"payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7\" (UID: \"f4084d5e-45e6-4acf-93d5-a9a9f2150b67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"
Sep 10 23:14:12.815031 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.814877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5541bf40-af13-42a3-89d3-0c79e2a2b35b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv\" (UID: \"5541bf40-af13-42a3-89d3-0c79e2a2b35b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"
Sep 10 23:14:12.815031 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.814895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q62dg\" (UniqueName: \"kubernetes.io/projected/5541bf40-af13-42a3-89d3-0c79e2a2b35b-kube-api-access-q62dg\") pod \"payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv\" (UID: \"5541bf40-af13-42a3-89d3-0c79e2a2b35b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"
Sep 10 23:14:12.815031 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.814921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4084d5e-45e6-4acf-93d5-a9a9f2150b67-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7\" (UID: \"f4084d5e-45e6-4acf-93d5-a9a9f2150b67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"
Sep 10 23:14:12.815588 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.815569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5541bf40-af13-42a3-89d3-0c79e2a2b35b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv\" (UID: \"5541bf40-af13-42a3-89d3-0c79e2a2b35b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"
Sep 10 23:14:12.815672 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.815655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4084d5e-45e6-4acf-93d5-a9a9f2150b67-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7\" (UID: \"f4084d5e-45e6-4acf-93d5-a9a9f2150b67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"
Sep 10 23:14:12.822943 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.822923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q62dg\" (UniqueName: \"kubernetes.io/projected/5541bf40-af13-42a3-89d3-0c79e2a2b35b-kube-api-access-q62dg\") pod \"payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv\" (UID: \"5541bf40-af13-42a3-89d3-0c79e2a2b35b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"
Sep 10 23:14:12.823283 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.823267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg4ln\" (UniqueName: \"kubernetes.io/projected/f4084d5e-45e6-4acf-93d5-a9a9f2150b67-kube-api-access-tg4ln\") pod \"payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7\" (UID: \"f4084d5e-45e6-4acf-93d5-a9a9f2150b67\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"
Sep 10 23:14:12.909089 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.909010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"
Sep 10 23:14:12.947476 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.947439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"
Sep 10 23:14:12.983474 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.983435    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7855-a41ca6"]
Sep 10 23:14:12.985838 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:12.985796    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7855-a41ca6"]
Sep 10 23:14:13.095845 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.095806    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7855-cb2d01"]
Sep 10 23:14:13.096204 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.096189    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ca9996-398f-4845-9b4b-905ca6665f35" containerName="test-curl-7855-a41ca6"
Sep 10 23:14:13.096255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.096205    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ca9996-398f-4845-9b4b-905ca6665f35" containerName="test-curl-7855-a41ca6"
Sep 10 23:14:13.096295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.096290    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ca9996-398f-4845-9b4b-905ca6665f35" containerName="test-curl-7855-a41ca6"
Sep 10 23:14:13.100279 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.100261    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7855-cb2d01"
Sep 10 23:14:13.103907 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.103876    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7855-cb2d01"]
Sep 10 23:14:13.186817 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.186789    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"]
Sep 10 23:14:13.188906 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:14:13.188870    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4084d5e_45e6_4acf_93d5_a9a9f2150b67.slice/crio-d4457ce58646953bf432fc15e49a32d66fc1c59b0abd633bafd4a97bf92c60ca WatchSource:0}: Error finding container d4457ce58646953bf432fc15e49a32d66fc1c59b0abd633bafd4a97bf92c60ca: Status 404 returned error can't find the container with id d4457ce58646953bf432fc15e49a32d66fc1c59b0abd633bafd4a97bf92c60ca
Sep 10 23:14:13.219101 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.219069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4f6h\" (UniqueName: \"kubernetes.io/projected/7435055f-56be-4484-8449-c1818e7ade02-kube-api-access-b4f6h\") pod \"test-curl-7855-cb2d01\" (UID: \"7435055f-56be-4484-8449-c1818e7ade02\") " pod="opendatahub/test-curl-7855-cb2d01"
Sep 10 23:14:13.239848 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.239820    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"]
Sep 10 23:14:13.243605 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:14:13.243576    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5541bf40_af13_42a3_89d3_0c79e2a2b35b.slice/crio-6a56610e4f546eabd127c4ab459edd5988d23271ff7cc7a030cbdf2182c97bae WatchSource:0}: Error finding container 6a56610e4f546eabd127c4ab459edd5988d23271ff7cc7a030cbdf2182c97bae: Status 404 returned error can't find the container with id 6a56610e4f546eabd127c4ab459edd5988d23271ff7cc7a030cbdf2182c97bae
Sep 10 23:14:13.319842 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.319810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4f6h\" (UniqueName: \"kubernetes.io/projected/7435055f-56be-4484-8449-c1818e7ade02-kube-api-access-b4f6h\") pod \"test-curl-7855-cb2d01\" (UID: \"7435055f-56be-4484-8449-c1818e7ade02\") " pod="opendatahub/test-curl-7855-cb2d01"
Sep 10 23:14:13.327688 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.327655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4f6h\" (UniqueName: \"kubernetes.io/projected/7435055f-56be-4484-8449-c1818e7ade02-kube-api-access-b4f6h\") pod \"test-curl-7855-cb2d01\" (UID: \"7435055f-56be-4484-8449-c1818e7ade02\") " pod="opendatahub/test-curl-7855-cb2d01"
Sep 10 23:14:13.389727 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.389694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv" event={"ID":"5541bf40-af13-42a3-89d3-0c79e2a2b35b","Type":"ContainerStarted","Data":"c13a844db1f9139f7a945c75d20d9997a276ce606e094cb1b22600e4ca62b2cf"}
Sep 10 23:14:13.390237 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.389750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv" event={"ID":"5541bf40-af13-42a3-89d3-0c79e2a2b35b","Type":"ContainerStarted","Data":"6a56610e4f546eabd127c4ab459edd5988d23271ff7cc7a030cbdf2182c97bae"}
Sep 10 23:14:13.390237 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.389814    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"
Sep 10 23:14:13.391481 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.391435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7" event={"ID":"f4084d5e-45e6-4acf-93d5-a9a9f2150b67","Type":"ContainerStarted","Data":"e14fb6d58c1174d532c8028059a6b033d0f2c992ea0115f22556fbcea1ec0adf"}
Sep 10 23:14:13.391613 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.391510    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7" event={"ID":"f4084d5e-45e6-4acf-93d5-a9a9f2150b67","Type":"ContainerStarted","Data":"d4457ce58646953bf432fc15e49a32d66fc1c59b0abd633bafd4a97bf92c60ca"}
Sep 10 23:14:13.391613 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.391609    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"
Sep 10 23:14:13.391747 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:14:13.391673    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7855-a41ca6" container="test-curl-7855-a41ca6"
Sep 10 23:14:13.408399 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.408349    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv" podStartSLOduration=1.408323099 podStartE2EDuration="1.408323099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:13.407361429 +0000 UTC m=+774.098938636" watchObservedRunningTime="2026-09-10 23:14:13.408323099 +0000 UTC m=+774.099900283"
Sep 10 23:14:13.410431 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.410412    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7855-cb2d01"
Sep 10 23:14:13.424335 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.424286    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7" podStartSLOduration=1.424269634 podStartE2EDuration="1.424269634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:13.423032617 +0000 UTC m=+774.114609815" watchObservedRunningTime="2026-09-10 23:14:13.424269634 +0000 UTC m=+774.115846818"
Sep 10 23:14:13.643091 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.643050    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"]
Sep 10 23:14:13.648473 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.648449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"
Sep 10 23:14:13.651331 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.651305    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-679c94-dockercfg-8wxmx\""
Sep 10 23:14:13.651536 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.651487    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-679c94\""
Sep 10 23:14:13.655893 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.655868    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"]
Sep 10 23:14:13.686868 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.686800    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7855-cb2d01"]
Sep 10 23:14:13.689306 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:14:13.689282    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7435055f_56be_4484_8449_c1818e7ade02.slice/crio-f7199829cd48fa6be6c1013261f08922a77937dcd8abaf5cb194d18a31fd0ff0 WatchSource:0}: Error finding container f7199829cd48fa6be6c1013261f08922a77937dcd8abaf5cb194d18a31fd0ff0: Status 404 returned error can't find the container with id f7199829cd48fa6be6c1013261f08922a77937dcd8abaf5cb194d18a31fd0ff0
Sep 10 23:14:13.724140 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.724104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvwkf\" (UniqueName: \"kubernetes.io/projected/8f8835f2-b94d-44ab-8283-d4e750300326-kube-api-access-dvwkf\") pod \"payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw\" (UID: \"8f8835f2-b94d-44ab-8283-d4e750300326\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"
Sep 10 23:14:13.724273 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.724242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8835f2-b94d-44ab-8283-d4e750300326-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw\" (UID: \"8f8835f2-b94d-44ab-8283-d4e750300326\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"
Sep 10 23:14:13.787443 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.787404    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"]
Sep 10 23:14:13.790996 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.790972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"
Sep 10 23:14:13.801807 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.801778    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"]
Sep 10 23:14:13.825248 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.825220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvwkf\" (UniqueName: \"kubernetes.io/projected/8f8835f2-b94d-44ab-8283-d4e750300326-kube-api-access-dvwkf\") pod \"payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw\" (UID: \"8f8835f2-b94d-44ab-8283-d4e750300326\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"
Sep 10 23:14:13.825413 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.825255    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl\" (UID: \"ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"
Sep 10 23:14:13.825413 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.825323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8835f2-b94d-44ab-8283-d4e750300326-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw\" (UID: \"8f8835f2-b94d-44ab-8283-d4e750300326\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"
Sep 10 23:14:13.825413 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.825364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrr44\" (UniqueName: \"kubernetes.io/projected/ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a-kube-api-access-mrr44\") pod \"payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl\" (UID: \"ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"
Sep 10 23:14:13.826188 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.826165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8835f2-b94d-44ab-8283-d4e750300326-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw\" (UID: \"8f8835f2-b94d-44ab-8283-d4e750300326\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"
Sep 10 23:14:13.833976 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.833952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvwkf\" (UniqueName: \"kubernetes.io/projected/8f8835f2-b94d-44ab-8283-d4e750300326-kube-api-access-dvwkf\") pod \"payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw\" (UID: \"8f8835f2-b94d-44ab-8283-d4e750300326\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"
Sep 10 23:14:13.926193 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.926143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrr44\" (UniqueName: \"kubernetes.io/projected/ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a-kube-api-access-mrr44\") pod \"payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl\" (UID: \"ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"
Sep 10 23:14:13.926403 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.926222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl\" (UID: \"ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"
Sep 10 23:14:13.927074 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.927051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl\" (UID: \"ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"
Sep 10 23:14:13.934661 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.934638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrr44\" (UniqueName: \"kubernetes.io/projected/ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a-kube-api-access-mrr44\") pod \"payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl\" (UID: \"ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"
Sep 10 23:14:13.962597 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:13.962570    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"
Sep 10 23:14:14.101931 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.101887    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"
Sep 10 23:14:14.239532 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.239483    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"]
Sep 10 23:14:14.241270 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:14:14.241243    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8835f2_b94d_44ab_8283_d4e750300326.slice/crio-df88f352a6135218cc9adb0dc732ac5d270c72934690d1e81d94be223e42a721 WatchSource:0}: Error finding container df88f352a6135218cc9adb0dc732ac5d270c72934690d1e81d94be223e42a721: Status 404 returned error can't find the container with id df88f352a6135218cc9adb0dc732ac5d270c72934690d1e81d94be223e42a721
Sep 10 23:14:14.384282 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.384252    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"]
Sep 10 23:14:14.386782 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:14:14.386739    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff15a1be_b7e3_4ceb_9c98_a5b75cfbbd0a.slice/crio-a15e86a063e79256180f20e9e2492a28c29bcc6921f42100e2256998f81970ae WatchSource:0}: Error finding container a15e86a063e79256180f20e9e2492a28c29bcc6921f42100e2256998f81970ae: Status 404 returned error can't find the container with id a15e86a063e79256180f20e9e2492a28c29bcc6921f42100e2256998f81970ae
Sep 10 23:14:14.397508 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.397462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw" event={"ID":"8f8835f2-b94d-44ab-8283-d4e750300326","Type":"ContainerStarted","Data":"99deed1b1635aa5be34b5c5cd52b4f6cf9363bfa5613c900b5a4129796499ad2"}
Sep 10 23:14:14.397897 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.397633    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"
Sep 10 23:14:14.397897 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.397665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw" event={"ID":"8f8835f2-b94d-44ab-8283-d4e750300326","Type":"ContainerStarted","Data":"df88f352a6135218cc9adb0dc732ac5d270c72934690d1e81d94be223e42a721"}
Sep 10 23:14:14.398798 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.398775    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl" event={"ID":"ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a","Type":"ContainerStarted","Data":"a15e86a063e79256180f20e9e2492a28c29bcc6921f42100e2256998f81970ae"}
Sep 10 23:14:14.400296 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.400274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7855-cb2d01" event={"ID":"7435055f-56be-4484-8449-c1818e7ade02","Type":"ContainerStarted","Data":"5b260c225aadcedd2af6f268bd4fd4338ab37f1f7be2857e42bbef3373ef130e"}
Sep 10 23:14:14.400390 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.400307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7855-cb2d01" event={"ID":"7435055f-56be-4484-8449-c1818e7ade02","Type":"ContainerStarted","Data":"f7199829cd48fa6be6c1013261f08922a77937dcd8abaf5cb194d18a31fd0ff0"}
Sep 10 23:14:14.400594 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.400554    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7855-a41ca6" podUID="88ca9996-398f-4845-9b4b-905ca6665f35" containerName="test-curl-7855-a41ca6" containerID="cri-o://856faded591bdeb2f79224054f5aa8470b1b0aa79bf8d367bef8562f2120268f" gracePeriod=2
Sep 10 23:14:14.415383 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.415336    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw" podStartSLOduration=1.415317005 podStartE2EDuration="1.415317005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:14.414534873 +0000 UTC m=+775.106112081" watchObservedRunningTime="2026-09-10 23:14:14.415317005 +0000 UTC m=+775.106894190"
Sep 10 23:14:14.429338 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.429291    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7855-cb2d01" podStartSLOduration=1.429275179 podStartE2EDuration="1.429275179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:14.42759651 +0000 UTC m=+775.119173697" watchObservedRunningTime="2026-09-10 23:14:14.429275179 +0000 UTC m=+775.120852362"
Sep 10 23:14:14.810003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.809961    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"]
Sep 10 23:14:14.813452 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.813430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"
Sep 10 23:14:14.815844 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.815821    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-6ce17e\""
Sep 10 23:14:14.815998 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.815840    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-6ce17e-dockercfg-7m6ls\""
Sep 10 23:14:14.825624 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.825594    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"]
Sep 10 23:14:14.828687 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.828449    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"]
Sep 10 23:14:14.846845 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.833600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"
Sep 10 23:14:14.865122 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.864362    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"]
Sep 10 23:14:14.935775 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.935732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bb1063-d926-4635-8724-e585ecbc5285-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8\" (UID: \"f0bb1063-d926-4635-8724-e585ecbc5285\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"
Sep 10 23:14:14.935967 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.935786    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv8ww\" (UniqueName: \"kubernetes.io/projected/f0bb1063-d926-4635-8724-e585ecbc5285-kube-api-access-rv8ww\") pod \"payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8\" (UID: \"f0bb1063-d926-4635-8724-e585ecbc5285\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"
Sep 10 23:14:14.935967 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.935870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5smk\" (UniqueName: \"kubernetes.io/projected/018d3164-a9ab-42dc-b2a9-3ecb5349fe73-kube-api-access-r5smk\") pod \"payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx\" (UID: \"018d3164-a9ab-42dc-b2a9-3ecb5349fe73\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"
Sep 10 23:14:14.936084 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:14.935973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/018d3164-a9ab-42dc-b2a9-3ecb5349fe73-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx\" (UID: \"018d3164-a9ab-42dc-b2a9-3ecb5349fe73\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"
Sep 10 23:14:15.037100 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.037049    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bb1063-d926-4635-8724-e585ecbc5285-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8\" (UID: \"f0bb1063-d926-4635-8724-e585ecbc5285\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"
Sep 10 23:14:15.037100 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.037099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv8ww\" (UniqueName: \"kubernetes.io/projected/f0bb1063-d926-4635-8724-e585ecbc5285-kube-api-access-rv8ww\") pod \"payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8\" (UID: \"f0bb1063-d926-4635-8724-e585ecbc5285\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"
Sep 10 23:14:15.037355 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.037140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5smk\" (UniqueName: \"kubernetes.io/projected/018d3164-a9ab-42dc-b2a9-3ecb5349fe73-kube-api-access-r5smk\") pod \"payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx\" (UID: \"018d3164-a9ab-42dc-b2a9-3ecb5349fe73\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"
Sep 10 23:14:15.037355 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.037211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/018d3164-a9ab-42dc-b2a9-3ecb5349fe73-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx\" (UID: \"018d3164-a9ab-42dc-b2a9-3ecb5349fe73\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"
Sep 10 23:14:15.038024 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.037997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bb1063-d926-4635-8724-e585ecbc5285-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8\" (UID: \"f0bb1063-d926-4635-8724-e585ecbc5285\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"
Sep 10 23:14:15.038128 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.038046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/018d3164-a9ab-42dc-b2a9-3ecb5349fe73-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx\" (UID: \"018d3164-a9ab-42dc-b2a9-3ecb5349fe73\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"
Sep 10 23:14:15.046127 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.046104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5smk\" (UniqueName: \"kubernetes.io/projected/018d3164-a9ab-42dc-b2a9-3ecb5349fe73-kube-api-access-r5smk\") pod \"payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx\" (UID: \"018d3164-a9ab-42dc-b2a9-3ecb5349fe73\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"
Sep 10 23:14:15.046239 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.046197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv8ww\" (UniqueName: \"kubernetes.io/projected/f0bb1063-d926-4635-8724-e585ecbc5285-kube-api-access-rv8ww\") pod \"payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8\" (UID: \"f0bb1063-d926-4635-8724-e585ecbc5285\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"
Sep 10 23:14:15.056920 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.056889    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7855-cb2d01"]
Sep 10 23:14:15.060581 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.060546    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7855-cb2d01"]
Sep 10 23:14:15.134160 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.134070    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"
Sep 10 23:14:15.172104 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.172046    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"
Sep 10 23:14:15.409411 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.409090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl" event={"ID":"ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a","Type":"ContainerStarted","Data":"f2d8719ab9e923b032a694f6709f87eae5b3b4d9edd51fe60d476089d052b9d6"}
Sep 10 23:14:15.409411 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:14:15.409313    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7855-cb2d01" container="test-curl-7855-cb2d01"
Sep 10 23:14:15.409411 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.409399    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"
Sep 10 23:14:15.417628 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.417602    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"]
Sep 10 23:14:15.420129 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:14:15.420107    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018d3164_a9ab_42dc_b2a9_3ecb5349fe73.slice/crio-64895eb1318b91982924b6a92fd284051aa0338ad415fca7be38ddc1e49ae064 WatchSource:0}: Error finding container 64895eb1318b91982924b6a92fd284051aa0338ad415fca7be38ddc1e49ae064: Status 404 returned error can't find the container with id 64895eb1318b91982924b6a92fd284051aa0338ad415fca7be38ddc1e49ae064
Sep 10 23:14:15.424881 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.424833    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl" podStartSLOduration=2.424815266 podStartE2EDuration="2.424815266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:15.424406035 +0000 UTC m=+776.115983247" watchObservedRunningTime="2026-09-10 23:14:15.424815266 +0000 UTC m=+776.116392456"
Sep 10 23:14:15.455776 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:15.455750    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"]
Sep 10 23:14:15.458101 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:14:15.458076    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bb1063_d926_4635_8724_e585ecbc5285.slice/crio-a6ff8b4baecb76f656e7ea6cafcd82685aef4eb969766321c48d718cba3c4722 WatchSource:0}: Error finding container a6ff8b4baecb76f656e7ea6cafcd82685aef4eb969766321c48d718cba3c4722: Status 404 returned error can't find the container with id a6ff8b4baecb76f656e7ea6cafcd82685aef4eb969766321c48d718cba3c4722
Sep 10 23:14:16.416125 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:16.416094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx" event={"ID":"018d3164-a9ab-42dc-b2a9-3ecb5349fe73","Type":"ContainerStarted","Data":"a3916a60a8cd0f441b9bee6059a2bbd1e124363eb33819f4e5930f22978921a1"}
Sep 10 23:14:16.416462 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:16.416155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx" event={"ID":"018d3164-a9ab-42dc-b2a9-3ecb5349fe73","Type":"ContainerStarted","Data":"64895eb1318b91982924b6a92fd284051aa0338ad415fca7be38ddc1e49ae064"}
Sep 10 23:14:16.416462 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:16.416184    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"
Sep 10 23:14:16.417661 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:16.417635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8" event={"ID":"f0bb1063-d926-4635-8724-e585ecbc5285","Type":"ContainerStarted","Data":"7cb91729343bfaa5e129b80f1ae6b315f1730149d405a3c03a1aca62c51aef27"}
Sep 10 23:14:16.417763 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:16.417676    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8" event={"ID":"f0bb1063-d926-4635-8724-e585ecbc5285","Type":"ContainerStarted","Data":"a6ff8b4baecb76f656e7ea6cafcd82685aef4eb969766321c48d718cba3c4722"}
Sep 10 23:14:16.417870 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:16.417848    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"
Sep 10 23:14:16.417947 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:16.417851    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7855-cb2d01" podUID="7435055f-56be-4484-8449-c1818e7ade02" containerName="test-curl-7855-cb2d01" containerID="cri-o://5b260c225aadcedd2af6f268bd4fd4338ab37f1f7be2857e42bbef3373ef130e" gracePeriod=2
Sep 10 23:14:16.432402 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:16.432361    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx" podStartSLOduration=2.4323475439999998 podStartE2EDuration="2.432347544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:16.431371216 +0000 UTC m=+777.122948419" watchObservedRunningTime="2026-09-10 23:14:16.432347544 +0000 UTC m=+777.123924724"
Sep 10 23:14:16.450165 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:16.450119    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8" podStartSLOduration=2.450102405 podStartE2EDuration="2.450102405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:16.448245582 +0000 UTC m=+777.139822785" watchObservedRunningTime="2026-09-10 23:14:16.450102405 +0000 UTC m=+777.141679589"
Sep 10 23:14:16.578358 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:16.578331    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7855-a41ca6"
Sep 10 23:14:16.652477 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:16.652441    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz28d\" (UniqueName: \"kubernetes.io/projected/88ca9996-398f-4845-9b4b-905ca6665f35-kube-api-access-lz28d\") pod \"88ca9996-398f-4845-9b4b-905ca6665f35\" (UID: \"88ca9996-398f-4845-9b4b-905ca6665f35\") "
Sep 10 23:14:16.654662 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:16.654602    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ca9996-398f-4845-9b4b-905ca6665f35-kube-api-access-lz28d" (OuterVolumeSpecName: "kube-api-access-lz28d") pod "88ca9996-398f-4845-9b4b-905ca6665f35" (UID: "88ca9996-398f-4845-9b4b-905ca6665f35"). InnerVolumeSpecName "kube-api-access-lz28d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:14:16.753348 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:16.753307    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz28d\" (UniqueName: \"kubernetes.io/projected/88ca9996-398f-4845-9b4b-905ca6665f35-kube-api-access-lz28d\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:14:17.424972 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:17.424940    2563 generic.go:356] "Generic (PLEG): container finished" podID="88ca9996-398f-4845-9b4b-905ca6665f35" containerID="856faded591bdeb2f79224054f5aa8470b1b0aa79bf8d367bef8562f2120268f" exitCode=137
Sep 10 23:14:17.425423 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:17.424993    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7855-a41ca6"
Sep 10 23:14:17.425423 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:17.425043    2563 scope.go:117] "RemoveContainer" containerID="856faded591bdeb2f79224054f5aa8470b1b0aa79bf8d367bef8562f2120268f"
Sep 10 23:14:17.433827 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:17.433805    2563 scope.go:117] "RemoveContainer" containerID="856faded591bdeb2f79224054f5aa8470b1b0aa79bf8d367bef8562f2120268f"
Sep 10 23:14:17.434144 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:14:17.434124    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856faded591bdeb2f79224054f5aa8470b1b0aa79bf8d367bef8562f2120268f\": container with ID starting with 856faded591bdeb2f79224054f5aa8470b1b0aa79bf8d367bef8562f2120268f not found: ID does not exist" containerID="856faded591bdeb2f79224054f5aa8470b1b0aa79bf8d367bef8562f2120268f"
Sep 10 23:14:17.434236 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:17.434150    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856faded591bdeb2f79224054f5aa8470b1b0aa79bf8d367bef8562f2120268f"} err="failed to get container status \"856faded591bdeb2f79224054f5aa8470b1b0aa79bf8d367bef8562f2120268f\": rpc error: code = NotFound desc = could not find container \"856faded591bdeb2f79224054f5aa8470b1b0aa79bf8d367bef8562f2120268f\": container with ID starting with 856faded591bdeb2f79224054f5aa8470b1b0aa79bf8d367bef8562f2120268f not found: ID does not exist"
Sep 10 23:14:17.438098 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:14:17.438073    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7855-a41ca6\" is forbidden: User \"system:node:ip-10-0-130-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-246.ec2.internal' and this object" podUID="88ca9996-398f-4845-9b4b-905ca6665f35" pod="opendatahub/test-curl-7855-a41ca6"
Sep 10 23:14:17.814731 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:17.814701    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88ca9996-398f-4845-9b4b-905ca6665f35" path="/var/lib/kubelet/pods/88ca9996-398f-4845-9b4b-905ca6665f35/volumes"
Sep 10 23:14:18.613527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:18.613482    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7855-cb2d01"
Sep 10 23:14:18.670688 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:18.670590    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4f6h\" (UniqueName: \"kubernetes.io/projected/7435055f-56be-4484-8449-c1818e7ade02-kube-api-access-b4f6h\") pod \"7435055f-56be-4484-8449-c1818e7ade02\" (UID: \"7435055f-56be-4484-8449-c1818e7ade02\") "
Sep 10 23:14:18.672724 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:18.672688    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7435055f-56be-4484-8449-c1818e7ade02-kube-api-access-b4f6h" (OuterVolumeSpecName: "kube-api-access-b4f6h") pod "7435055f-56be-4484-8449-c1818e7ade02" (UID: "7435055f-56be-4484-8449-c1818e7ade02"). InnerVolumeSpecName "kube-api-access-b4f6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:14:18.772060 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:18.772019    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4f6h\" (UniqueName: \"kubernetes.io/projected/7435055f-56be-4484-8449-c1818e7ade02-kube-api-access-b4f6h\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:14:19.437101 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:19.437068    2563 generic.go:356] "Generic (PLEG): container finished" podID="7435055f-56be-4484-8449-c1818e7ade02" containerID="5b260c225aadcedd2af6f268bd4fd4338ab37f1f7be2857e42bbef3373ef130e" exitCode=137
Sep 10 23:14:19.437276 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:19.437128    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7855-cb2d01"
Sep 10 23:14:19.437276 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:19.437165    2563 scope.go:117] "RemoveContainer" containerID="5b260c225aadcedd2af6f268bd4fd4338ab37f1f7be2857e42bbef3373ef130e"
Sep 10 23:14:19.445581 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:19.445564    2563 scope.go:117] "RemoveContainer" containerID="5b260c225aadcedd2af6f268bd4fd4338ab37f1f7be2857e42bbef3373ef130e"
Sep 10 23:14:19.445850 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:14:19.445831    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b260c225aadcedd2af6f268bd4fd4338ab37f1f7be2857e42bbef3373ef130e\": container with ID starting with 5b260c225aadcedd2af6f268bd4fd4338ab37f1f7be2857e42bbef3373ef130e not found: ID does not exist" containerID="5b260c225aadcedd2af6f268bd4fd4338ab37f1f7be2857e42bbef3373ef130e"
Sep 10 23:14:19.445915 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:19.445856    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b260c225aadcedd2af6f268bd4fd4338ab37f1f7be2857e42bbef3373ef130e"} err="failed to get container status \"5b260c225aadcedd2af6f268bd4fd4338ab37f1f7be2857e42bbef3373ef130e\": rpc error: code = NotFound desc = could not find container \"5b260c225aadcedd2af6f268bd4fd4338ab37f1f7be2857e42bbef3373ef130e\": container with ID starting with 5b260c225aadcedd2af6f268bd4fd4338ab37f1f7be2857e42bbef3373ef130e not found: ID does not exist"
Sep 10 23:14:19.447226 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:14:19.447206    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7855-cb2d01\" is forbidden: User \"system:node:ip-10-0-130-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-246.ec2.internal' and this object" podUID="7435055f-56be-4484-8449-c1818e7ade02" pod="opendatahub/test-curl-7855-cb2d01"
Sep 10 23:14:19.813402 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:14:19.813362    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7855-cb2d01\" is forbidden: User \"system:node:ip-10-0-130-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-246.ec2.internal' and this object" podUID="7435055f-56be-4484-8449-c1818e7ade02" pod="opendatahub/test-curl-7855-cb2d01"
Sep 10 23:14:19.817768 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:19.817738    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7435055f-56be-4484-8449-c1818e7ade02" path="/var/lib/kubelet/pods/7435055f-56be-4484-8449-c1818e7ade02/volumes"
Sep 10 23:14:20.672547 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.672484    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"]
Sep 10 23:14:20.672933 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.672919    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7435055f-56be-4484-8449-c1818e7ade02" containerName="test-curl-7855-cb2d01"
Sep 10 23:14:20.673003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.672935    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7435055f-56be-4484-8449-c1818e7ade02" containerName="test-curl-7855-cb2d01"
Sep 10 23:14:20.673045 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.673002    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7435055f-56be-4484-8449-c1818e7ade02" containerName="test-curl-7855-cb2d01"
Sep 10 23:14:20.677604 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.677585    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"
Sep 10 23:14:20.679909 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.679883    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-689fe3\""
Sep 10 23:14:20.680042 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.680001    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-689fe3-dockercfg-bjcvh\""
Sep 10 23:14:20.685376 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.685060    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"]
Sep 10 23:14:20.763987 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.763955    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"]
Sep 10 23:14:20.767559 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.767537    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"
Sep 10 23:14:20.781515 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.781475    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"]
Sep 10 23:14:20.793801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.793776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f639f0-5a65-4aea-a802-cb9fa8f76f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt\" (UID: \"98f639f0-5a65-4aea-a802-cb9fa8f76f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"
Sep 10 23:14:20.793923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.793815    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwjzw\" (UniqueName: \"kubernetes.io/projected/98f639f0-5a65-4aea-a802-cb9fa8f76f4c-kube-api-access-zwjzw\") pod \"payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt\" (UID: \"98f639f0-5a65-4aea-a802-cb9fa8f76f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"
Sep 10 23:14:20.894546 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.894470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f639f0-5a65-4aea-a802-cb9fa8f76f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt\" (UID: \"98f639f0-5a65-4aea-a802-cb9fa8f76f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"
Sep 10 23:14:20.895014 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.894585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwjzw\" (UniqueName: \"kubernetes.io/projected/98f639f0-5a65-4aea-a802-cb9fa8f76f4c-kube-api-access-zwjzw\") pod \"payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt\" (UID: \"98f639f0-5a65-4aea-a802-cb9fa8f76f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"
Sep 10 23:14:20.895014 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.894631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae41f1ae-4afe-46d1-8081-71163b580cec-plugins-config-volume\") pod \"payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf\" (UID: \"ae41f1ae-4afe-46d1-8081-71163b580cec\") " pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"
Sep 10 23:14:20.895014 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.894921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ndrq\" (UniqueName: \"kubernetes.io/projected/ae41f1ae-4afe-46d1-8081-71163b580cec-kube-api-access-5ndrq\") pod \"payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf\" (UID: \"ae41f1ae-4afe-46d1-8081-71163b580cec\") " pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"
Sep 10 23:14:20.895301 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.895281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f639f0-5a65-4aea-a802-cb9fa8f76f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt\" (UID: \"98f639f0-5a65-4aea-a802-cb9fa8f76f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"
Sep 10 23:14:20.902729 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.902710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwjzw\" (UniqueName: \"kubernetes.io/projected/98f639f0-5a65-4aea-a802-cb9fa8f76f4c-kube-api-access-zwjzw\") pod \"payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt\" (UID: \"98f639f0-5a65-4aea-a802-cb9fa8f76f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"
Sep 10 23:14:20.988852 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.988754    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"
Sep 10 23:14:20.996344 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.996310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndrq\" (UniqueName: \"kubernetes.io/projected/ae41f1ae-4afe-46d1-8081-71163b580cec-kube-api-access-5ndrq\") pod \"payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf\" (UID: \"ae41f1ae-4afe-46d1-8081-71163b580cec\") " pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"
Sep 10 23:14:20.996534 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.996382    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae41f1ae-4afe-46d1-8081-71163b580cec-plugins-config-volume\") pod \"payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf\" (UID: \"ae41f1ae-4afe-46d1-8081-71163b580cec\") " pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"
Sep 10 23:14:20.997112 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:20.997093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae41f1ae-4afe-46d1-8081-71163b580cec-plugins-config-volume\") pod \"payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf\" (UID: \"ae41f1ae-4afe-46d1-8081-71163b580cec\") " pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"
Sep 10 23:14:21.015948 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.015916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ndrq\" (UniqueName: \"kubernetes.io/projected/ae41f1ae-4afe-46d1-8081-71163b580cec-kube-api-access-5ndrq\") pod \"payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf\" (UID: \"ae41f1ae-4afe-46d1-8081-71163b580cec\") " pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"
Sep 10 23:14:21.077899 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.077865    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"
Sep 10 23:14:21.263238 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.263181    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"]
Sep 10 23:14:21.265501 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:14:21.265464    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f639f0_5a65_4aea_a802_cb9fa8f76f4c.slice/crio-b352ec6727b706a7f13c3eb7a3cf5cb016ebafa8125b4dd9f17308f5003e932f WatchSource:0}: Error finding container b352ec6727b706a7f13c3eb7a3cf5cb016ebafa8125b4dd9f17308f5003e932f: Status 404 returned error can't find the container with id b352ec6727b706a7f13c3eb7a3cf5cb016ebafa8125b4dd9f17308f5003e932f
Sep 10 23:14:21.359080 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.359053    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"]
Sep 10 23:14:21.360755 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:14:21.360728    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae41f1ae_4afe_46d1_8081_71163b580cec.slice/crio-1cfc62e33752c6cd24a263228af952ce67cc992caed36dd7cc99b97a3167b8c9 WatchSource:0}: Error finding container 1cfc62e33752c6cd24a263228af952ce67cc992caed36dd7cc99b97a3167b8c9: Status 404 returned error can't find the container with id 1cfc62e33752c6cd24a263228af952ce67cc992caed36dd7cc99b97a3167b8c9
Sep 10 23:14:21.449088 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.449046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt" event={"ID":"98f639f0-5a65-4aea-a802-cb9fa8f76f4c","Type":"ContainerStarted","Data":"f1bfd978d873ab96bcb647867bd37d8f48a6d506321dbfdf4a04e5c399fcc4a8"}
Sep 10 23:14:21.449244 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.449107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt" event={"ID":"98f639f0-5a65-4aea-a802-cb9fa8f76f4c","Type":"ContainerStarted","Data":"b352ec6727b706a7f13c3eb7a3cf5cb016ebafa8125b4dd9f17308f5003e932f"}
Sep 10 23:14:21.449244 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.449238    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"
Sep 10 23:14:21.450766 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.450737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf" event={"ID":"ae41f1ae-4afe-46d1-8081-71163b580cec","Type":"ContainerStarted","Data":"1082444bccffad3f3d04b3404baaa993bededa29ceeb7f3b0a7f456e3196efc6"}
Sep 10 23:14:21.450882 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.450782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf" event={"ID":"ae41f1ae-4afe-46d1-8081-71163b580cec","Type":"ContainerStarted","Data":"1cfc62e33752c6cd24a263228af952ce67cc992caed36dd7cc99b97a3167b8c9"}
Sep 10 23:14:21.450882 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.450810    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"
Sep 10 23:14:21.470595 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.470542    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt" podStartSLOduration=1.470526314 podStartE2EDuration="1.470526314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:21.468422139 +0000 UTC m=+782.159999348" watchObservedRunningTime="2026-09-10 23:14:21.470526314 +0000 UTC m=+782.162103499"
Sep 10 23:14:21.486680 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.486626    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf" podStartSLOduration=1.486606974 podStartE2EDuration="1.486606974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:21.485376447 +0000 UTC m=+782.176953646" watchObservedRunningTime="2026-09-10 23:14:21.486606974 +0000 UTC m=+782.178184162"
Sep 10 23:14:21.746307 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.746265    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"]
Sep 10 23:14:21.751077 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.751045    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"
Sep 10 23:14:21.753676 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.753641    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f897c8bb\""
Sep 10 23:14:21.759724 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.759698    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"]
Sep 10 23:14:21.782727 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.782695    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"]
Sep 10 23:14:21.786192 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.786172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"
Sep 10 23:14:21.788896 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.788872    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f897c8bb-dockercfg-nxdqf\""
Sep 10 23:14:21.800171 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.800113    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"]
Sep 10 23:14:21.803645 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.803618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8px8b\" (UniqueName: \"kubernetes.io/projected/ff2389d5-1f03-45b5-818c-cca94f667a57-kube-api-access-8px8b\") pod \"payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt\" (UID: \"ff2389d5-1f03-45b5-818c-cca94f667a57\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"
Sep 10 23:14:21.803774 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.803674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2389d5-1f03-45b5-818c-cca94f667a57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt\" (UID: \"ff2389d5-1f03-45b5-818c-cca94f667a57\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"
Sep 10 23:14:21.905173 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.905132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2389d5-1f03-45b5-818c-cca94f667a57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt\" (UID: \"ff2389d5-1f03-45b5-818c-cca94f667a57\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"
Sep 10 23:14:21.905173 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.905187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8px8b\" (UniqueName: \"kubernetes.io/projected/ff2389d5-1f03-45b5-818c-cca94f667a57-kube-api-access-8px8b\") pod \"payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt\" (UID: \"ff2389d5-1f03-45b5-818c-cca94f667a57\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"
Sep 10 23:14:21.905634 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.905238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ea0a76-34ac-4f69-98da-4c398a10e065-plugins-config-volume\") pod \"payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s\" (UID: \"31ea0a76-34ac-4f69-98da-4c398a10e065\") " pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"
Sep 10 23:14:21.905634 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.905259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdllb\" (UniqueName: \"kubernetes.io/projected/31ea0a76-34ac-4f69-98da-4c398a10e065-kube-api-access-mdllb\") pod \"payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s\" (UID: \"31ea0a76-34ac-4f69-98da-4c398a10e065\") " pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"
Sep 10 23:14:21.906123 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.906098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2389d5-1f03-45b5-818c-cca94f667a57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt\" (UID: \"ff2389d5-1f03-45b5-818c-cca94f667a57\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"
Sep 10 23:14:21.913563 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:21.913534    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8px8b\" (UniqueName: \"kubernetes.io/projected/ff2389d5-1f03-45b5-818c-cca94f667a57-kube-api-access-8px8b\") pod \"payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt\" (UID: \"ff2389d5-1f03-45b5-818c-cca94f667a57\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"
Sep 10 23:14:22.006306 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:22.006201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdllb\" (UniqueName: \"kubernetes.io/projected/31ea0a76-34ac-4f69-98da-4c398a10e065-kube-api-access-mdllb\") pod \"payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s\" (UID: \"31ea0a76-34ac-4f69-98da-4c398a10e065\") " pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"
Sep 10 23:14:22.006474 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:22.006354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ea0a76-34ac-4f69-98da-4c398a10e065-plugins-config-volume\") pod \"payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s\" (UID: \"31ea0a76-34ac-4f69-98da-4c398a10e065\") " pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"
Sep 10 23:14:22.007258 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:22.007237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ea0a76-34ac-4f69-98da-4c398a10e065-plugins-config-volume\") pod \"payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s\" (UID: \"31ea0a76-34ac-4f69-98da-4c398a10e065\") " pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"
Sep 10 23:14:22.014016 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:22.013989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdllb\" (UniqueName: \"kubernetes.io/projected/31ea0a76-34ac-4f69-98da-4c398a10e065-kube-api-access-mdllb\") pod \"payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s\" (UID: \"31ea0a76-34ac-4f69-98da-4c398a10e065\") " pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"
Sep 10 23:14:22.067299 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:22.067235    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"
Sep 10 23:14:22.106438 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:22.106400    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"
Sep 10 23:14:22.363125 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:22.363094    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"]
Sep 10 23:14:22.365870 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:14:22.365844    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2389d5_1f03_45b5_818c_cca94f667a57.slice/crio-40996ff31988d3234e4ae9877ce2d6cf76e7e75037bd1ddfb4b394b3be727da3 WatchSource:0}: Error finding container 40996ff31988d3234e4ae9877ce2d6cf76e7e75037bd1ddfb4b394b3be727da3: Status 404 returned error can't find the container with id 40996ff31988d3234e4ae9877ce2d6cf76e7e75037bd1ddfb4b394b3be727da3
Sep 10 23:14:22.410709 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:22.410684    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"]
Sep 10 23:14:22.412249 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:14:22.412223    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ea0a76_34ac_4f69_98da_4c398a10e065.slice/crio-dbbf0eda024854b2097d4467475d3de02324b9a1a11ffa67baa380910b7e937e WatchSource:0}: Error finding container dbbf0eda024854b2097d4467475d3de02324b9a1a11ffa67baa380910b7e937e: Status 404 returned error can't find the container with id dbbf0eda024854b2097d4467475d3de02324b9a1a11ffa67baa380910b7e937e
Sep 10 23:14:22.458337 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:22.458265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s" event={"ID":"31ea0a76-34ac-4f69-98da-4c398a10e065","Type":"ContainerStarted","Data":"dbbf0eda024854b2097d4467475d3de02324b9a1a11ffa67baa380910b7e937e"}
Sep 10 23:14:22.461423 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:22.461082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt" event={"ID":"ff2389d5-1f03-45b5-818c-cca94f667a57","Type":"ContainerStarted","Data":"718afedab015274d5b86cff0c4eb70f0cd4867e6ca3cfe917ecfea5971c0a9c8"}
Sep 10 23:14:22.461423 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:22.461143    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt" event={"ID":"ff2389d5-1f03-45b5-818c-cca94f667a57","Type":"ContainerStarted","Data":"40996ff31988d3234e4ae9877ce2d6cf76e7e75037bd1ddfb4b394b3be727da3"}
Sep 10 23:14:22.461423 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:22.461271    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"
Sep 10 23:14:22.479225 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:22.478584    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt" podStartSLOduration=1.478561364 podStartE2EDuration="1.478561364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:22.478012776 +0000 UTC m=+783.169589988" watchObservedRunningTime="2026-09-10 23:14:22.478561364 +0000 UTC m=+783.170138549"
Sep 10 23:14:23.469562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:23.469528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s" event={"ID":"31ea0a76-34ac-4f69-98da-4c398a10e065","Type":"ContainerStarted","Data":"85d09e99a494a77b4f443c6d69bf1922e7df8cd41d887d91e9be13f2cb1cea78"}
Sep 10 23:14:23.470033 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:23.469732    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"
Sep 10 23:14:23.487241 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:23.487185    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s" podStartSLOduration=2.487169101 podStartE2EDuration="2.487169101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:23.486453264 +0000 UTC m=+784.178030473" watchObservedRunningTime="2026-09-10 23:14:23.487169101 +0000 UTC m=+784.178746284"
Sep 10 23:14:24.402454 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:24.402421    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"
Sep 10 23:14:24.402747 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:24.402553    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"
Sep 10 23:14:25.410971 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:25.410932    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"
Sep 10 23:14:26.419954 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:26.419919    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"
Sep 10 23:14:27.427670 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:27.427639    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"
Sep 10 23:14:27.428154 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:27.427724    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"
Sep 10 23:14:32.463593 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:32.463554    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"
Sep 10 23:14:32.463997 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:32.463723    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"
Sep 10 23:14:33.472296 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:33.472203    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"
Sep 10 23:14:34.477339 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:34.477306    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"
Sep 10 23:14:42.759984 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:42.759904    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"]
Sep 10 23:14:42.764670 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:42.764652    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"
Sep 10 23:14:42.767179 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:42.767157    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-568856-dockercfg-4sgps\""
Sep 10 23:14:42.767179 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:42.767162    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-568856\""
Sep 10 23:14:42.780603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:42.780567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"]
Sep 10 23:14:42.908850 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:42.908817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ceb34f-b641-4ee5-a4a8-a8e27625f150-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8\" (UID: \"64ceb34f-b641-4ee5-a4a8-a8e27625f150\") " pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"
Sep 10 23:14:42.909024 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:42.908918    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf5pm\" (UniqueName: \"kubernetes.io/projected/64ceb34f-b641-4ee5-a4a8-a8e27625f150-kube-api-access-sf5pm\") pod \"payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8\" (UID: \"64ceb34f-b641-4ee5-a4a8-a8e27625f150\") " pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"
Sep 10 23:14:43.010066 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:43.009981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf5pm\" (UniqueName: \"kubernetes.io/projected/64ceb34f-b641-4ee5-a4a8-a8e27625f150-kube-api-access-sf5pm\") pod \"payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8\" (UID: \"64ceb34f-b641-4ee5-a4a8-a8e27625f150\") " pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"
Sep 10 23:14:43.010222 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:43.010148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ceb34f-b641-4ee5-a4a8-a8e27625f150-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8\" (UID: \"64ceb34f-b641-4ee5-a4a8-a8e27625f150\") " pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"
Sep 10 23:14:43.010989 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:43.010962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ceb34f-b641-4ee5-a4a8-a8e27625f150-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8\" (UID: \"64ceb34f-b641-4ee5-a4a8-a8e27625f150\") " pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"
Sep 10 23:14:43.017667 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:43.017635    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf5pm\" (UniqueName: \"kubernetes.io/projected/64ceb34f-b641-4ee5-a4a8-a8e27625f150-kube-api-access-sf5pm\") pod \"payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8\" (UID: \"64ceb34f-b641-4ee5-a4a8-a8e27625f150\") " pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"
Sep 10 23:14:43.074957 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:43.074927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"
Sep 10 23:14:43.348430 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:43.348404    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"]
Sep 10 23:14:43.350405 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:14:43.350378    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ceb34f_b641_4ee5_a4a8_a8e27625f150.slice/crio-2a66013f7ddf795414748bb3132c7d587319af675c855cccf5b44cdb66827ced WatchSource:0}: Error finding container 2a66013f7ddf795414748bb3132c7d587319af675c855cccf5b44cdb66827ced: Status 404 returned error can't find the container with id 2a66013f7ddf795414748bb3132c7d587319af675c855cccf5b44cdb66827ced
Sep 10 23:14:43.579375 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:43.579335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8" event={"ID":"64ceb34f-b641-4ee5-a4a8-a8e27625f150","Type":"ContainerStarted","Data":"ad09103f85270b8b48fda448f2a7724de42130a405477d406d120f3064b5f34d"}
Sep 10 23:14:43.579583 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:43.579440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8" event={"ID":"64ceb34f-b641-4ee5-a4a8-a8e27625f150","Type":"ContainerStarted","Data":"2a66013f7ddf795414748bb3132c7d587319af675c855cccf5b44cdb66827ced"}
Sep 10 23:14:43.579583 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:43.579482    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"
Sep 10 23:14:43.598408 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:43.598308    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8" podStartSLOduration=1.598292588 podStartE2EDuration="1.598292588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:43.596061884 +0000 UTC m=+804.287639096" watchObservedRunningTime="2026-09-10 23:14:43.598292588 +0000 UTC m=+804.289869773"
Sep 10 23:14:49.830427 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:49.830390    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq"]
Sep 10 23:14:49.835943 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:49.835924    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq"
Sep 10 23:14:49.838166 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:49.838138    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:14:49.838166 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:49.838160    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:14:49.842874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:49.842842    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq"]
Sep 10 23:14:49.871543 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:49.871479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/622e5cf9-d046-4280-b964-ec7c73061a5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq\" (UID: \"622e5cf9-d046-4280-b964-ec7c73061a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq"
Sep 10 23:14:49.972208 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:49.972177    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/622e5cf9-d046-4280-b964-ec7c73061a5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq\" (UID: \"622e5cf9-d046-4280-b964-ec7c73061a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq"
Sep 10 23:14:49.972846 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:49.972820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/622e5cf9-d046-4280-b964-ec7c73061a5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq\" (UID: \"622e5cf9-d046-4280-b964-ec7c73061a5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq"
Sep 10 23:14:50.149894 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:50.149802    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq"
Sep 10 23:14:50.424183 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:50.424155    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq"]
Sep 10 23:14:50.426912 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:14:50.426882    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622e5cf9_d046_4280_b964_ec7c73061a5c.slice/crio-6bbe981b33457f986312a0806704b85693f9b35a9d26312ea705866f3fc69eb1 WatchSource:0}: Error finding container 6bbe981b33457f986312a0806704b85693f9b35a9d26312ea705866f3fc69eb1: Status 404 returned error can't find the container with id 6bbe981b33457f986312a0806704b85693f9b35a9d26312ea705866f3fc69eb1
Sep 10 23:14:50.617022 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:50.616986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq" event={"ID":"622e5cf9-d046-4280-b964-ec7c73061a5c","Type":"ContainerStarted","Data":"6bbe981b33457f986312a0806704b85693f9b35a9d26312ea705866f3fc69eb1"}
Sep 10 23:14:53.635884 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:53.635852    2563 generic.go:356] "Generic (PLEG): container finished" podID="622e5cf9-d046-4280-b964-ec7c73061a5c" containerID="f0904e08fd606123ca44c7bd1cd382afc6dcc397dfc587f81127de82d82ae3da" exitCode=0
Sep 10 23:14:53.636294 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:53.635940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq" event={"ID":"622e5cf9-d046-4280-b964-ec7c73061a5c","Type":"ContainerDied","Data":"f0904e08fd606123ca44c7bd1cd382afc6dcc397dfc587f81127de82d82ae3da"}
Sep 10 23:14:54.586514 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:54.586453    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"
Sep 10 23:14:54.782670 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:54.782643    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq"
Sep 10 23:14:54.822229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:54.822200    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/622e5cf9-d046-4280-b964-ec7c73061a5c-maas-api-service-ca\") pod \"622e5cf9-d046-4280-b964-ec7c73061a5c\" (UID: \"622e5cf9-d046-4280-b964-ec7c73061a5c\") "
Sep 10 23:14:54.822563 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:54.822540    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/622e5cf9-d046-4280-b964-ec7c73061a5c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "622e5cf9-d046-4280-b964-ec7c73061a5c" (UID: "622e5cf9-d046-4280-b964-ec7c73061a5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:14:54.923735 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:54.923657    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/622e5cf9-d046-4280-b964-ec7c73061a5c-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:14:55.654461 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:55.654420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq" event={"ID":"622e5cf9-d046-4280-b964-ec7c73061a5c","Type":"ContainerDied","Data":"6bbe981b33457f986312a0806704b85693f9b35a9d26312ea705866f3fc69eb1"}
Sep 10 23:14:55.654461 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:55.654453    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq"
Sep 10 23:14:55.654682 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:14:55.654520    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bbe981b33457f986312a0806704b85693f9b35a9d26312ea705866f3fc69eb1"
Sep 10 23:15:00.184772 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.184730    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-689fe3-29818035-tcg82"]
Sep 10 23:15:00.185212 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.185198    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="622e5cf9-d046-4280-b964-ec7c73061a5c" containerName="revoke-keys"
Sep 10 23:15:00.185212 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.185214    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="622e5cf9-d046-4280-b964-ec7c73061a5c" containerName="revoke-keys"
Sep 10 23:15:00.185295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.185290    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="622e5cf9-d046-4280-b964-ec7c73061a5c" containerName="revoke-keys"
Sep 10 23:15:00.189688 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.189670    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-689fe3-29818035-tcg82"
Sep 10 23:15:00.196473 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.196151    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:15:00.199987 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.199953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-689fe3-29818035-tcg82"
Sep 10 23:15:00.201969 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.201942    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818035-d7ld4"]
Sep 10 23:15:00.206008 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.205986    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818035-d7ld4"
Sep 10 23:15:00.208971 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.208948    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7025b9-29818035-jhwml"]
Sep 10 23:15:00.216700 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.216062    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7025b9-29818035-jhwml"
Sep 10 23:15:00.216700 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.216345    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-689fe3-29818035-tcg82"]
Sep 10 23:15:00.226256 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.226189    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818035-d7ld4"]
Sep 10 23:15:00.238589 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.238559    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7025b9-29818035-jhwml"]
Sep 10 23:15:00.244656 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.243465    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f897c8bb-29818035-9rtkf"]
Sep 10 23:15:00.254282 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.254059    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818035-d7ld4"
Sep 10 23:15:00.255799 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.255773    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f897c8bb-29818035-9rtkf"]
Sep 10 23:15:00.255922 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.255891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f897c8bb-29818035-9rtkf"
Sep 10 23:15:00.268384 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.268354    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7025b9-29818035-jhwml"
Sep 10 23:15:00.317308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.316479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f897c8bb-29818035-9rtkf"
Sep 10 23:15:00.522745 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.522694    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-689fe3-29818035-tcg82"]
Sep 10 23:15:00.525132 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:15:00.525101    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144e80a1_38e0_4c86_85ee_222e0d249112.slice/crio-c773d6ec821a18a9b2966ba122c6e77d6d05e7c68dd16d2a2acbc16335eb9cd6 WatchSource:0}: Error finding container c773d6ec821a18a9b2966ba122c6e77d6d05e7c68dd16d2a2acbc16335eb9cd6: Status 404 returned error can't find the container with id c773d6ec821a18a9b2966ba122c6e77d6d05e7c68dd16d2a2acbc16335eb9cd6
Sep 10 23:15:00.550155 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.550131    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818035-d7ld4"]
Sep 10 23:15:00.559606 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:15:00.559581    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec78084d_b577_44c4_9a7d_e9a3f2b19bde.slice/crio-59e03a91b074554c5befaacabec62a3637efda8b1922a1d7604d2e3fff6cdb1d WatchSource:0}: Error finding container 59e03a91b074554c5befaacabec62a3637efda8b1922a1d7604d2e3fff6cdb1d: Status 404 returned error can't find the container with id 59e03a91b074554c5befaacabec62a3637efda8b1922a1d7604d2e3fff6cdb1d
Sep 10 23:15:00.563137 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.563049    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7025b9-29818035-jhwml"]
Sep 10 23:15:00.564707 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:15:00.564677    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b218306_51e7_4e3c_b2c0_416191435dec.slice/crio-9d68cfb6324e9b002c8d62f4265af5bd555c97889c2cff7dd76cd3a20e666a8b WatchSource:0}: Error finding container 9d68cfb6324e9b002c8d62f4265af5bd555c97889c2cff7dd76cd3a20e666a8b: Status 404 returned error can't find the container with id 9d68cfb6324e9b002c8d62f4265af5bd555c97889c2cff7dd76cd3a20e666a8b
Sep 10 23:15:00.609205 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.609178    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f897c8bb-29818035-9rtkf"]
Sep 10 23:15:00.611316 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:15:00.611294    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab8363c_9bea_48f6_9b7b_9a2de0b1486a.slice/crio-3ceb36f58eb2d04ab04c64e23b73ba9fdcd379edb9e38e813ed453e9cd90050d WatchSource:0}: Error finding container 3ceb36f58eb2d04ab04c64e23b73ba9fdcd379edb9e38e813ed453e9cd90050d: Status 404 returned error can't find the container with id 3ceb36f58eb2d04ab04c64e23b73ba9fdcd379edb9e38e813ed453e9cd90050d
Sep 10 23:15:00.681689 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.681651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f897c8bb-29818035-9rtkf" event={"ID":"3ab8363c-9bea-48f6-9b7b-9a2de0b1486a","Type":"ContainerStarted","Data":"3ceb36f58eb2d04ab04c64e23b73ba9fdcd379edb9e38e813ed453e9cd90050d"}
Sep 10 23:15:00.682888 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.682857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818035-d7ld4" event={"ID":"ec78084d-b577-44c4-9a7d-e9a3f2b19bde","Type":"ContainerStarted","Data":"59e03a91b074554c5befaacabec62a3637efda8b1922a1d7604d2e3fff6cdb1d"}
Sep 10 23:15:00.684023 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.684001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7025b9-29818035-jhwml" event={"ID":"7b218306-51e7-4e3c-b2c0-416191435dec","Type":"ContainerStarted","Data":"9d68cfb6324e9b002c8d62f4265af5bd555c97889c2cff7dd76cd3a20e666a8b"}
Sep 10 23:15:00.685154 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:00.685136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-689fe3-29818035-tcg82" event={"ID":"144e80a1-38e0-4c86-85ee-222e0d249112","Type":"ContainerStarted","Data":"c773d6ec821a18a9b2966ba122c6e77d6d05e7c68dd16d2a2acbc16335eb9cd6"}
Sep 10 23:15:01.127234 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.127150    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f897c8bb-29818035-9rtkf"]
Sep 10 23:15:01.163636 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.163317    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"]
Sep 10 23:15:01.164340 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.164084    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s" podUID="31ea0a76-34ac-4f69-98da-4c398a10e065" containerName="payload-processing" containerID="cri-o://85d09e99a494a77b4f443c6d69bf1922e7df8cd41d887d91e9be13f2cb1cea78" gracePeriod=30
Sep 10 23:15:01.165958 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.165934    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"]
Sep 10 23:15:01.166145 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.166122    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt" podUID="ff2389d5-1f03-45b5-818c-cca94f667a57" containerName="payload-pre-processing" containerID="cri-o://718afedab015274d5b86cff0c4eb70f0cd4867e6ca3cfe917ecfea5971c0a9c8" gracePeriod=30
Sep 10 23:15:01.329582 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.329516    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq"]
Sep 10 23:15:01.331699 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.331667    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f897c8bb-ljbvq"]
Sep 10 23:15:01.444077 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.444047    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"
Sep 10 23:15:01.474251 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.474220    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"
Sep 10 23:15:01.484271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.484238    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2389d5-1f03-45b5-818c-cca94f667a57-plugins-config-volume\") pod \"ff2389d5-1f03-45b5-818c-cca94f667a57\" (UID: \"ff2389d5-1f03-45b5-818c-cca94f667a57\") "
Sep 10 23:15:01.484486 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.484410    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8px8b\" (UniqueName: \"kubernetes.io/projected/ff2389d5-1f03-45b5-818c-cca94f667a57-kube-api-access-8px8b\") pod \"ff2389d5-1f03-45b5-818c-cca94f667a57\" (UID: \"ff2389d5-1f03-45b5-818c-cca94f667a57\") "
Sep 10 23:15:01.484644 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.484609    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2389d5-1f03-45b5-818c-cca94f667a57-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff2389d5-1f03-45b5-818c-cca94f667a57" (UID: "ff2389d5-1f03-45b5-818c-cca94f667a57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:15:01.484746 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.484716    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2389d5-1f03-45b5-818c-cca94f667a57-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:15:01.486596 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.486567    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2389d5-1f03-45b5-818c-cca94f667a57-kube-api-access-8px8b" (OuterVolumeSpecName: "kube-api-access-8px8b") pod "ff2389d5-1f03-45b5-818c-cca94f667a57" (UID: "ff2389d5-1f03-45b5-818c-cca94f667a57"). InnerVolumeSpecName "kube-api-access-8px8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:15:01.585892 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.585858    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ea0a76-34ac-4f69-98da-4c398a10e065-plugins-config-volume\") pod \"31ea0a76-34ac-4f69-98da-4c398a10e065\" (UID: \"31ea0a76-34ac-4f69-98da-4c398a10e065\") "
Sep 10 23:15:01.586024 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.585904    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdllb\" (UniqueName: \"kubernetes.io/projected/31ea0a76-34ac-4f69-98da-4c398a10e065-kube-api-access-mdllb\") pod \"31ea0a76-34ac-4f69-98da-4c398a10e065\" (UID: \"31ea0a76-34ac-4f69-98da-4c398a10e065\") "
Sep 10 23:15:01.586095 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.586084    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8px8b\" (UniqueName: \"kubernetes.io/projected/ff2389d5-1f03-45b5-818c-cca94f667a57-kube-api-access-8px8b\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:15:01.586232 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.586205    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ea0a76-34ac-4f69-98da-4c398a10e065-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31ea0a76-34ac-4f69-98da-4c398a10e065" (UID: "31ea0a76-34ac-4f69-98da-4c398a10e065"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:15:01.637969 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.637931    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ea0a76-34ac-4f69-98da-4c398a10e065-kube-api-access-mdllb" (OuterVolumeSpecName: "kube-api-access-mdllb") pod "31ea0a76-34ac-4f69-98da-4c398a10e065" (UID: "31ea0a76-34ac-4f69-98da-4c398a10e065"). InnerVolumeSpecName "kube-api-access-mdllb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:15:01.687011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.686981    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31ea0a76-34ac-4f69-98da-4c398a10e065-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:15:01.687011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.687007    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdllb\" (UniqueName: \"kubernetes.io/projected/31ea0a76-34ac-4f69-98da-4c398a10e065-kube-api-access-mdllb\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:15:01.691624 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.691590    2563 generic.go:356] "Generic (PLEG): container finished" podID="7b218306-51e7-4e3c-b2c0-416191435dec" containerID="11d17288e6f509420fcff9736145d4dc69ce518646b649999ab60f66de16d5da" exitCode=0
Sep 10 23:15:01.691773 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.691706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7025b9-29818035-jhwml" event={"ID":"7b218306-51e7-4e3c-b2c0-416191435dec","Type":"ContainerDied","Data":"11d17288e6f509420fcff9736145d4dc69ce518646b649999ab60f66de16d5da"}
Sep 10 23:15:01.693483 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.693460    2563 generic.go:356] "Generic (PLEG): container finished" podID="144e80a1-38e0-4c86-85ee-222e0d249112" containerID="23602df0388f16adfe5b025775b6c76109bb5c4f9c75a6a812d3abcedef2e9bf" exitCode=0
Sep 10 23:15:01.693617 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.693542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-689fe3-29818035-tcg82" event={"ID":"144e80a1-38e0-4c86-85ee-222e0d249112","Type":"ContainerDied","Data":"23602df0388f16adfe5b025775b6c76109bb5c4f9c75a6a812d3abcedef2e9bf"}
Sep 10 23:15:01.694820 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.694801    2563 generic.go:356] "Generic (PLEG): container finished" podID="31ea0a76-34ac-4f69-98da-4c398a10e065" containerID="85d09e99a494a77b4f443c6d69bf1922e7df8cd41d887d91e9be13f2cb1cea78" exitCode=0
Sep 10 23:15:01.694918 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.694843    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"
Sep 10 23:15:01.694918 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.694881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s" event={"ID":"31ea0a76-34ac-4f69-98da-4c398a10e065","Type":"ContainerDied","Data":"85d09e99a494a77b4f443c6d69bf1922e7df8cd41d887d91e9be13f2cb1cea78"}
Sep 10 23:15:01.695009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.694928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s" event={"ID":"31ea0a76-34ac-4f69-98da-4c398a10e065","Type":"ContainerDied","Data":"dbbf0eda024854b2097d4467475d3de02324b9a1a11ffa67baa380910b7e937e"}
Sep 10 23:15:01.695009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.694962    2563 scope.go:117] "RemoveContainer" containerID="85d09e99a494a77b4f443c6d69bf1922e7df8cd41d887d91e9be13f2cb1cea78"
Sep 10 23:15:01.696589 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.696571    2563 generic.go:356] "Generic (PLEG): container finished" podID="3ab8363c-9bea-48f6-9b7b-9a2de0b1486a" containerID="7ffb06a3a97b28a64d626e5be12f330efbc489d499a0278f6a03265eda662bdc" exitCode=6
Sep 10 23:15:01.696697 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.696633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f897c8bb-29818035-9rtkf" event={"ID":"3ab8363c-9bea-48f6-9b7b-9a2de0b1486a","Type":"ContainerDied","Data":"7ffb06a3a97b28a64d626e5be12f330efbc489d499a0278f6a03265eda662bdc"}
Sep 10 23:15:01.696697 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.696676    2563 scope.go:117] "RemoveContainer" containerID="7ffb06a3a97b28a64d626e5be12f330efbc489d499a0278f6a03265eda662bdc"
Sep 10 23:15:01.698006 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.697986    2563 generic.go:356] "Generic (PLEG): container finished" podID="ff2389d5-1f03-45b5-818c-cca94f667a57" containerID="718afedab015274d5b86cff0c4eb70f0cd4867e6ca3cfe917ecfea5971c0a9c8" exitCode=0
Sep 10 23:15:01.698119 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.698036    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"
Sep 10 23:15:01.698119 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.698078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt" event={"ID":"ff2389d5-1f03-45b5-818c-cca94f667a57","Type":"ContainerDied","Data":"718afedab015274d5b86cff0c4eb70f0cd4867e6ca3cfe917ecfea5971c0a9c8"}
Sep 10 23:15:01.698119 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.698116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt" event={"ID":"ff2389d5-1f03-45b5-818c-cca94f667a57","Type":"ContainerDied","Data":"40996ff31988d3234e4ae9877ce2d6cf76e7e75037bd1ddfb4b394b3be727da3"}
Sep 10 23:15:01.699963 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.699912    2563 generic.go:356] "Generic (PLEG): container finished" podID="ec78084d-b577-44c4-9a7d-e9a3f2b19bde" containerID="0177ba5483ce4ffd8a5b09c27fc48f4aa0409efcb55e251db48a9ba4f56c6701" exitCode=0
Sep 10 23:15:01.699963 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.699947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818035-d7ld4" event={"ID":"ec78084d-b577-44c4-9a7d-e9a3f2b19bde","Type":"ContainerDied","Data":"0177ba5483ce4ffd8a5b09c27fc48f4aa0409efcb55e251db48a9ba4f56c6701"}
Sep 10 23:15:01.705898 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.705681    2563 scope.go:117] "RemoveContainer" containerID="85d09e99a494a77b4f443c6d69bf1922e7df8cd41d887d91e9be13f2cb1cea78"
Sep 10 23:15:01.706826 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:15:01.706530    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d09e99a494a77b4f443c6d69bf1922e7df8cd41d887d91e9be13f2cb1cea78\": container with ID starting with 85d09e99a494a77b4f443c6d69bf1922e7df8cd41d887d91e9be13f2cb1cea78 not found: ID does not exist" containerID="85d09e99a494a77b4f443c6d69bf1922e7df8cd41d887d91e9be13f2cb1cea78"
Sep 10 23:15:01.706826 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.706646    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d09e99a494a77b4f443c6d69bf1922e7df8cd41d887d91e9be13f2cb1cea78"} err="failed to get container status \"85d09e99a494a77b4f443c6d69bf1922e7df8cd41d887d91e9be13f2cb1cea78\": rpc error: code = NotFound desc = could not find container \"85d09e99a494a77b4f443c6d69bf1922e7df8cd41d887d91e9be13f2cb1cea78\": container with ID starting with 85d09e99a494a77b4f443c6d69bf1922e7df8cd41d887d91e9be13f2cb1cea78 not found: ID does not exist"
Sep 10 23:15:01.706826 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.706680    2563 scope.go:117] "RemoveContainer" containerID="718afedab015274d5b86cff0c4eb70f0cd4867e6ca3cfe917ecfea5971c0a9c8"
Sep 10 23:15:01.715817 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.715798    2563 scope.go:117] "RemoveContainer" containerID="718afedab015274d5b86cff0c4eb70f0cd4867e6ca3cfe917ecfea5971c0a9c8"
Sep 10 23:15:01.716126 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:15:01.716104    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"718afedab015274d5b86cff0c4eb70f0cd4867e6ca3cfe917ecfea5971c0a9c8\": container with ID starting with 718afedab015274d5b86cff0c4eb70f0cd4867e6ca3cfe917ecfea5971c0a9c8 not found: ID does not exist" containerID="718afedab015274d5b86cff0c4eb70f0cd4867e6ca3cfe917ecfea5971c0a9c8"
Sep 10 23:15:01.716168 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.716139    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"718afedab015274d5b86cff0c4eb70f0cd4867e6ca3cfe917ecfea5971c0a9c8"} err="failed to get container status \"718afedab015274d5b86cff0c4eb70f0cd4867e6ca3cfe917ecfea5971c0a9c8\": rpc error: code = NotFound desc = could not find container \"718afedab015274d5b86cff0c4eb70f0cd4867e6ca3cfe917ecfea5971c0a9c8\": container with ID starting with 718afedab015274d5b86cff0c4eb70f0cd4867e6ca3cfe917ecfea5971c0a9c8 not found: ID does not exist"
Sep 10 23:15:01.767378 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.767342    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"]
Sep 10 23:15:01.774142 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.774111    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f897c8bb-8d9d6c597-j4c6s"]
Sep 10 23:15:01.784889 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.784855    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"]
Sep 10 23:15:01.786696 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.786675    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f897c8bb-86bdf55bb-ftptt"]
Sep 10 23:15:01.815240 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.815207    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ea0a76-34ac-4f69-98da-4c398a10e065" path="/var/lib/kubelet/pods/31ea0a76-34ac-4f69-98da-4c398a10e065/volumes"
Sep 10 23:15:01.815558 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.815545    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="622e5cf9-d046-4280-b964-ec7c73061a5c" path="/var/lib/kubelet/pods/622e5cf9-d046-4280-b964-ec7c73061a5c/volumes"
Sep 10 23:15:01.815874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:01.815862    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff2389d5-1f03-45b5-818c-cca94f667a57" path="/var/lib/kubelet/pods/ff2389d5-1f03-45b5-818c-cca94f667a57/volumes"
Sep 10 23:15:02.708330 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:02.708296    2563 generic.go:356] "Generic (PLEG): container finished" podID="3ab8363c-9bea-48f6-9b7b-9a2de0b1486a" containerID="3c62a4b3e288f36b06d902ce9c307779a3ec3a4417c1ee596c2af2021fb9cb80" exitCode=6
Sep 10 23:15:02.708793 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:02.708385    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f897c8bb-29818035-9rtkf" event={"ID":"3ab8363c-9bea-48f6-9b7b-9a2de0b1486a","Type":"ContainerDied","Data":"3c62a4b3e288f36b06d902ce9c307779a3ec3a4417c1ee596c2af2021fb9cb80"}
Sep 10 23:15:02.708793 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:02.708465    2563 scope.go:117] "RemoveContainer" containerID="7ffb06a3a97b28a64d626e5be12f330efbc489d499a0278f6a03265eda662bdc"
Sep 10 23:15:02.859135 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:02.859102    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f897c8bb-29818035-9rtkf"
Sep 10 23:15:02.903982 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:02.903955    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7025b9-29818035-jhwml"
Sep 10 23:15:02.926322 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:02.926298    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-689fe3-29818035-tcg82"
Sep 10 23:15:02.937056 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:02.937031    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818035-d7ld4"
Sep 10 23:15:03.720249 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.720215    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f897c8bb-29818035-9rtkf"
Sep 10 23:15:03.720249 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.720240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f897c8bb-29818035-9rtkf" event={"ID":"3ab8363c-9bea-48f6-9b7b-9a2de0b1486a","Type":"ContainerDied","Data":"3ceb36f58eb2d04ab04c64e23b73ba9fdcd379edb9e38e813ed453e9cd90050d"}
Sep 10 23:15:03.720781 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.720332    2563 scope.go:117] "RemoveContainer" containerID="3c62a4b3e288f36b06d902ce9c307779a3ec3a4417c1ee596c2af2021fb9cb80"
Sep 10 23:15:03.721884 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.721680    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818035-d7ld4"
Sep 10 23:15:03.721884 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.721715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818035-d7ld4" event={"ID":"ec78084d-b577-44c4-9a7d-e9a3f2b19bde","Type":"ContainerDied","Data":"59e03a91b074554c5befaacabec62a3637efda8b1922a1d7604d2e3fff6cdb1d"}
Sep 10 23:15:03.721884 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.721756    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59e03a91b074554c5befaacabec62a3637efda8b1922a1d7604d2e3fff6cdb1d"
Sep 10 23:15:03.723410 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.723380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7025b9-29818035-jhwml" event={"ID":"7b218306-51e7-4e3c-b2c0-416191435dec","Type":"ContainerDied","Data":"9d68cfb6324e9b002c8d62f4265af5bd555c97889c2cff7dd76cd3a20e666a8b"}
Sep 10 23:15:03.723538 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.723418    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7025b9-29818035-jhwml"
Sep 10 23:15:03.723538 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.723433    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d68cfb6324e9b002c8d62f4265af5bd555c97889c2cff7dd76cd3a20e666a8b"
Sep 10 23:15:03.724882 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.724853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-689fe3-29818035-tcg82" event={"ID":"144e80a1-38e0-4c86-85ee-222e0d249112","Type":"ContainerDied","Data":"c773d6ec821a18a9b2966ba122c6e77d6d05e7c68dd16d2a2acbc16335eb9cd6"}
Sep 10 23:15:03.725002 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.724901    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c773d6ec821a18a9b2966ba122c6e77d6d05e7c68dd16d2a2acbc16335eb9cd6"
Sep 10 23:15:03.725002 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.724904    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-689fe3-29818035-tcg82"
Sep 10 23:15:03.745925 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.745901    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f897c8bb-29818035-9rtkf"]
Sep 10 23:15:03.749303 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.749282    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f897c8bb-29818035-9rtkf"]
Sep 10 23:15:03.814025 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:03.813991    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ab8363c-9bea-48f6-9b7b-9a2de0b1486a" path="/var/lib/kubelet/pods/3ab8363c-9bea-48f6-9b7b-9a2de0b1486a/volumes"
Sep 10 23:15:06.764005 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.763969    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"]
Sep 10 23:15:06.764826 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.764792    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec78084d-b577-44c4-9a7d-e9a3f2b19bde" containerName="cleanup"
Sep 10 23:15:06.764826 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.764820    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec78084d-b577-44c4-9a7d-e9a3f2b19bde" containerName="cleanup"
Sep 10 23:15:06.765320 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765298    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff2389d5-1f03-45b5-818c-cca94f667a57" containerName="payload-pre-processing"
Sep 10 23:15:06.765403 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765366    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff2389d5-1f03-45b5-818c-cca94f667a57" containerName="payload-pre-processing"
Sep 10 23:15:06.765471 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765440    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ea0a76-34ac-4f69-98da-4c398a10e065" containerName="payload-processing"
Sep 10 23:15:06.765471 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765452    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ea0a76-34ac-4f69-98da-4c398a10e065" containerName="payload-processing"
Sep 10 23:15:06.765663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765475    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b218306-51e7-4e3c-b2c0-416191435dec" containerName="cleanup"
Sep 10 23:15:06.765663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765484    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b218306-51e7-4e3c-b2c0-416191435dec" containerName="cleanup"
Sep 10 23:15:06.765663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765543    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ab8363c-9bea-48f6-9b7b-9a2de0b1486a" containerName="cleanup"
Sep 10 23:15:06.765663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765554    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ab8363c-9bea-48f6-9b7b-9a2de0b1486a" containerName="cleanup"
Sep 10 23:15:06.765663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765584    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144e80a1-38e0-4c86-85ee-222e0d249112" containerName="cleanup"
Sep 10 23:15:06.765663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765593    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="144e80a1-38e0-4c86-85ee-222e0d249112" containerName="cleanup"
Sep 10 23:15:06.765663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765607    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ab8363c-9bea-48f6-9b7b-9a2de0b1486a" containerName="cleanup"
Sep 10 23:15:06.765663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765640    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ab8363c-9bea-48f6-9b7b-9a2de0b1486a" containerName="cleanup"
Sep 10 23:15:06.766015 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765962    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b218306-51e7-4e3c-b2c0-416191435dec" containerName="cleanup"
Sep 10 23:15:06.766015 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765985    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec78084d-b577-44c4-9a7d-e9a3f2b19bde" containerName="cleanup"
Sep 10 23:15:06.766015 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.765994    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ab8363c-9bea-48f6-9b7b-9a2de0b1486a" containerName="cleanup"
Sep 10 23:15:06.766015 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.766010    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff2389d5-1f03-45b5-818c-cca94f667a57" containerName="payload-pre-processing"
Sep 10 23:15:06.766198 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.766029    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="144e80a1-38e0-4c86-85ee-222e0d249112" containerName="cleanup"
Sep 10 23:15:06.766198 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.766040    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ea0a76-34ac-4f69-98da-4c398a10e065" containerName="payload-processing"
Sep 10 23:15:06.771664 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.771637    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"
Sep 10 23:15:06.774142 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.774111    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-7dbe617a-dockercfg-bvh8s\""
Sep 10 23:15:06.774314 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.774293    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7dbe617a\""
Sep 10 23:15:06.779854 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.779824    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"]
Sep 10 23:15:06.810552 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.810518    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"]
Sep 10 23:15:06.810992 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.810981    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ab8363c-9bea-48f6-9b7b-9a2de0b1486a" containerName="cleanup"
Sep 10 23:15:06.814705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.814680    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"
Sep 10 23:15:06.825015 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.824986    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"]
Sep 10 23:15:06.838790 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.838762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c68e804e-e181-401c-97a8-de5e91045f0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv\" (UID: \"c68e804e-e181-401c-97a8-de5e91045f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"
Sep 10 23:15:06.838942 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.838885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82shn\" (UniqueName: \"kubernetes.io/projected/c68e804e-e181-401c-97a8-de5e91045f0b-kube-api-access-82shn\") pod \"payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv\" (UID: \"c68e804e-e181-401c-97a8-de5e91045f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"
Sep 10 23:15:06.940190 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.940155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c68e804e-e181-401c-97a8-de5e91045f0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv\" (UID: \"c68e804e-e181-401c-97a8-de5e91045f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"
Sep 10 23:15:06.940384 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.940269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8v8g\" (UniqueName: \"kubernetes.io/projected/cb5d2f22-0dcc-457c-8931-f6f6fac769bd-kube-api-access-d8v8g\") pod \"payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q\" (UID: \"cb5d2f22-0dcc-457c-8931-f6f6fac769bd\") " pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"
Sep 10 23:15:06.940384 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.940356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5d2f22-0dcc-457c-8931-f6f6fac769bd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q\" (UID: \"cb5d2f22-0dcc-457c-8931-f6f6fac769bd\") " pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"
Sep 10 23:15:06.940532 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.940386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82shn\" (UniqueName: \"kubernetes.io/projected/c68e804e-e181-401c-97a8-de5e91045f0b-kube-api-access-82shn\") pod \"payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv\" (UID: \"c68e804e-e181-401c-97a8-de5e91045f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"
Sep 10 23:15:06.941221 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.941197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c68e804e-e181-401c-97a8-de5e91045f0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv\" (UID: \"c68e804e-e181-401c-97a8-de5e91045f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"
Sep 10 23:15:06.948754 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:06.948732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82shn\" (UniqueName: \"kubernetes.io/projected/c68e804e-e181-401c-97a8-de5e91045f0b-kube-api-access-82shn\") pod \"payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv\" (UID: \"c68e804e-e181-401c-97a8-de5e91045f0b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"
Sep 10 23:15:07.041757 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.041676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8v8g\" (UniqueName: \"kubernetes.io/projected/cb5d2f22-0dcc-457c-8931-f6f6fac769bd-kube-api-access-d8v8g\") pod \"payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q\" (UID: \"cb5d2f22-0dcc-457c-8931-f6f6fac769bd\") " pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"
Sep 10 23:15:07.041757 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.041736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5d2f22-0dcc-457c-8931-f6f6fac769bd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q\" (UID: \"cb5d2f22-0dcc-457c-8931-f6f6fac769bd\") " pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"
Sep 10 23:15:07.042407 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.042389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5d2f22-0dcc-457c-8931-f6f6fac769bd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q\" (UID: \"cb5d2f22-0dcc-457c-8931-f6f6fac769bd\") " pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"
Sep 10 23:15:07.050391 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.050362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8v8g\" (UniqueName: \"kubernetes.io/projected/cb5d2f22-0dcc-457c-8931-f6f6fac769bd-kube-api-access-d8v8g\") pod \"payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q\" (UID: \"cb5d2f22-0dcc-457c-8931-f6f6fac769bd\") " pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"
Sep 10 23:15:07.084695 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.084668    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"
Sep 10 23:15:07.126580 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.126545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"
Sep 10 23:15:07.356446 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.356416    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"]
Sep 10 23:15:07.357714 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:15:07.357691    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68e804e_e181_401c_97a8_de5e91045f0b.slice/crio-0327b80960fa791ab67624af0bcd10c7a952f5bbfb81215ab3d8f5125b566022 WatchSource:0}: Error finding container 0327b80960fa791ab67624af0bcd10c7a952f5bbfb81215ab3d8f5125b566022: Status 404 returned error can't find the container with id 0327b80960fa791ab67624af0bcd10c7a952f5bbfb81215ab3d8f5125b566022
Sep 10 23:15:07.415734 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.415712    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"]
Sep 10 23:15:07.417339 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:15:07.417310    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5d2f22_0dcc_457c_8931_f6f6fac769bd.slice/crio-d4e037cc37f4ff37db13b7f1ba941daa2bf21d43d80a77c523a4829eebf1a2d1 WatchSource:0}: Error finding container d4e037cc37f4ff37db13b7f1ba941daa2bf21d43d80a77c523a4829eebf1a2d1: Status 404 returned error can't find the container with id d4e037cc37f4ff37db13b7f1ba941daa2bf21d43d80a77c523a4829eebf1a2d1
Sep 10 23:15:07.738324 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.738229    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"]
Sep 10 23:15:07.742016 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.741995    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.744556 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.744530    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"openshift-service-ca.crt\""
Sep 10 23:15:07.744675 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.744563    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"kube-root-ca.crt\""
Sep 10 23:15:07.745765 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.745695    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"default-dockercfg-2lgbw\""
Sep 10 23:15:07.745765 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.745748    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"simulator-w4-7025b9-kserve-self-signed-certs\""
Sep 10 23:15:07.752203 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.752169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q" event={"ID":"cb5d2f22-0dcc-457c-8931-f6f6fac769bd","Type":"ContainerStarted","Data":"b1ef52d5d2951d2c8ecad86324d14dbfa9ff55860017a3463a4da31f39a70d96"}
Sep 10 23:15:07.752309 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.752225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q" event={"ID":"cb5d2f22-0dcc-457c-8931-f6f6fac769bd","Type":"ContainerStarted","Data":"d4e037cc37f4ff37db13b7f1ba941daa2bf21d43d80a77c523a4829eebf1a2d1"}
Sep 10 23:15:07.752811 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.752790    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"
Sep 10 23:15:07.753750 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.753727    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"]
Sep 10 23:15:07.754924 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.754900    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv" event={"ID":"c68e804e-e181-401c-97a8-de5e91045f0b","Type":"ContainerStarted","Data":"fbfc448d47cc9156624f885c0c559555a1f7b8ad2e2f33bc149db8264309c76a"}
Sep 10 23:15:07.755015 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.754948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv" event={"ID":"c68e804e-e181-401c-97a8-de5e91045f0b","Type":"ContainerStarted","Data":"0327b80960fa791ab67624af0bcd10c7a952f5bbfb81215ab3d8f5125b566022"}
Sep 10 23:15:07.755015 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.754976    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"
Sep 10 23:15:07.778466 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.778404    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv" podStartSLOduration=1.7783847910000001 podStartE2EDuration="1.778384791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:07.775383155 +0000 UTC m=+828.466960365" watchObservedRunningTime="2026-09-10 23:15:07.778384791 +0000 UTC m=+828.469961981"
Sep 10 23:15:07.791877 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.791825    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q" podStartSLOduration=1.791811292 podStartE2EDuration="1.791811292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:07.789991209 +0000 UTC m=+828.481568419" watchObservedRunningTime="2026-09-10 23:15:07.791811292 +0000 UTC m=+828.483388476"
Sep 10 23:15:07.849032 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.849000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-tmp-dir\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.849224 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.849107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-dshm\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.849224 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.849145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-model-cache\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.849343 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.849265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hghqq\" (UniqueName: \"kubernetes.io/projected/637941ab-24a5-4220-a8ab-4f62c974654e-kube-api-access-hghqq\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.849398 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.849350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-home\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.849455 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.849425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637941ab-24a5-4220-a8ab-4f62c974654e-tls-certs\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.950196 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.950158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-dshm\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.950391 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.950202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-model-cache\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.950391 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.950284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hghqq\" (UniqueName: \"kubernetes.io/projected/637941ab-24a5-4220-a8ab-4f62c974654e-kube-api-access-hghqq\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.950391 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.950346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-home\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.950586 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.950406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637941ab-24a5-4220-a8ab-4f62c974654e-tls-certs\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.950586 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.950463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-tmp-dir\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.950825 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.950699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-model-cache\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.950948 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.950888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-tmp-dir\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.951317 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.951127    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-home\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.953244 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.953220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-dshm\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.953635 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.953611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637941ab-24a5-4220-a8ab-4f62c974654e-tls-certs\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:07.959613 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:07.959580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hghqq\" (UniqueName: \"kubernetes.io/projected/637941ab-24a5-4220-a8ab-4f62c974654e-kube-api-access-hghqq\") pod \"simulator-w4-7025b9-kserve-c98c9d948-nndrq\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") " pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:08.053959 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:08.053871    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:08.366223 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:08.366180    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"]
Sep 10 23:15:08.761406 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:08.761316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq" event={"ID":"637941ab-24a5-4220-a8ab-4f62c974654e","Type":"ContainerStarted","Data":"9e523deb17d0d1a37cf4064c95139dbe6bc942f02e872193b61f21584e4804b1"}
Sep 10 23:15:08.761406 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:08.761363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq" event={"ID":"637941ab-24a5-4220-a8ab-4f62c974654e","Type":"ContainerStarted","Data":"71e61490a1d4a6bdc7ee0101d64c4509475343eee7a508344c6fd81e17133312"}
Sep 10 23:15:08.781210 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:08.781154    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq" podStartSLOduration=1.781137645 podStartE2EDuration="1.781137645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:08.778537547 +0000 UTC m=+829.470114756" watchObservedRunningTime="2026-09-10 23:15:08.781137645 +0000 UTC m=+829.472714829"
Sep 10 23:15:18.054624 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:18.054591    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:18.055147 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:18.054660    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:18.067152 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:18.067130    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:18.763520 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:18.763473    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"
Sep 10 23:15:18.827391 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:18.827361    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:15:19.769292 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:19.769257    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"
Sep 10 23:15:30.843979 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:30.843947    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr"]
Sep 10 23:15:30.851842 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:30.851817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr"
Sep 10 23:15:30.854515 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:30.854378    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:15:30.855243 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:30.855219    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:15:30.855870 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:30.855842    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr"]
Sep 10 23:15:30.970724 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:30.970690    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf0d31b-3556-4c79-9f24-e3f3112e50c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr\" (UID: \"abf0d31b-3556-4c79-9f24-e3f3112e50c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr"
Sep 10 23:15:31.071439 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:31.071407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf0d31b-3556-4c79-9f24-e3f3112e50c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr\" (UID: \"abf0d31b-3556-4c79-9f24-e3f3112e50c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr"
Sep 10 23:15:31.072086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:31.072062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf0d31b-3556-4c79-9f24-e3f3112e50c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr\" (UID: \"abf0d31b-3556-4c79-9f24-e3f3112e50c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr"
Sep 10 23:15:31.164373 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:31.164294    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr"
Sep 10 23:15:31.434685 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:31.434661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr"]
Sep 10 23:15:31.436557 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:15:31.436527    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf0d31b_3556_4c79_9f24_e3f3112e50c0.slice/crio-95929087ff040f05ddf2f799e3e43c3b4e6bf5ab696097c0c75a6f69caee96e0 WatchSource:0}: Error finding container 95929087ff040f05ddf2f799e3e43c3b4e6bf5ab696097c0c75a6f69caee96e0: Status 404 returned error can't find the container with id 95929087ff040f05ddf2f799e3e43c3b4e6bf5ab696097c0c75a6f69caee96e0
Sep 10 23:15:31.897983 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:31.897951    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr" event={"ID":"abf0d31b-3556-4c79-9f24-e3f3112e50c0","Type":"ContainerStarted","Data":"3ddf69411e8620110a0b8193401a5e2fdd14aac8ebcd205fccbdc112a2bd17f8"}
Sep 10 23:15:31.898350 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:31.897998    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr" event={"ID":"abf0d31b-3556-4c79-9f24-e3f3112e50c0","Type":"ContainerStarted","Data":"95929087ff040f05ddf2f799e3e43c3b4e6bf5ab696097c0c75a6f69caee96e0"}
Sep 10 23:15:31.915181 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:31.915133    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr" podStartSLOduration=1.915118002 podStartE2EDuration="1.915118002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:31.913949174 +0000 UTC m=+852.605526370" watchObservedRunningTime="2026-09-10 23:15:31.915118002 +0000 UTC m=+852.606695186"
Sep 10 23:15:32.904442 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:32.904408    2563 generic.go:356] "Generic (PLEG): container finished" podID="abf0d31b-3556-4c79-9f24-e3f3112e50c0" containerID="3ddf69411e8620110a0b8193401a5e2fdd14aac8ebcd205fccbdc112a2bd17f8" exitCode=0
Sep 10 23:15:32.904846 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:32.904502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr" event={"ID":"abf0d31b-3556-4c79-9f24-e3f3112e50c0","Type":"ContainerDied","Data":"3ddf69411e8620110a0b8193401a5e2fdd14aac8ebcd205fccbdc112a2bd17f8"}
Sep 10 23:15:34.053417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:34.053396    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr"
Sep 10 23:15:34.098154 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:34.098127    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf0d31b-3556-4c79-9f24-e3f3112e50c0-maas-api-service-ca\") pod \"abf0d31b-3556-4c79-9f24-e3f3112e50c0\" (UID: \"abf0d31b-3556-4c79-9f24-e3f3112e50c0\") "
Sep 10 23:15:34.098458 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:34.098436    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf0d31b-3556-4c79-9f24-e3f3112e50c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "abf0d31b-3556-4c79-9f24-e3f3112e50c0" (UID: "abf0d31b-3556-4c79-9f24-e3f3112e50c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:15:34.199270 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:34.199208    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abf0d31b-3556-4c79-9f24-e3f3112e50c0-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:15:34.916232 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:34.916206    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr"
Sep 10 23:15:34.916404 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:34.916252    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr" event={"ID":"abf0d31b-3556-4c79-9f24-e3f3112e50c0","Type":"ContainerDied","Data":"95929087ff040f05ddf2f799e3e43c3b4e6bf5ab696097c0c75a6f69caee96e0"}
Sep 10 23:15:34.916404 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:34.916301    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95929087ff040f05ddf2f799e3e43c3b4e6bf5ab696097c0c75a6f69caee96e0"
Sep 10 23:15:37.414701 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.414666    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"]
Sep 10 23:15:37.415219 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.414918    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q" podUID="cb5d2f22-0dcc-457c-8931-f6f6fac769bd" containerName="payload-processing" containerID="cri-o://b1ef52d5d2951d2c8ecad86324d14dbfa9ff55860017a3463a4da31f39a70d96" gracePeriod=30
Sep 10 23:15:37.432916 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.432884    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"]
Sep 10 23:15:37.433253 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.433110    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv" podUID="c68e804e-e181-401c-97a8-de5e91045f0b" containerName="payload-pre-processing" containerID="cri-o://fbfc448d47cc9156624f885c0c559555a1f7b8ad2e2f33bc149db8264309c76a" gracePeriod=30
Sep 10 23:15:37.577538 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.577503    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr"]
Sep 10 23:15:37.581079 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.580998    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7dbe617a-h7hwr"]
Sep 10 23:15:37.723990 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.723963    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"
Sep 10 23:15:37.726960 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.726938    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"
Sep 10 23:15:37.815086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.815051    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abf0d31b-3556-4c79-9f24-e3f3112e50c0" path="/var/lib/kubelet/pods/abf0d31b-3556-4c79-9f24-e3f3112e50c0/volumes"
Sep 10 23:15:37.833650 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.833621    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8v8g\" (UniqueName: \"kubernetes.io/projected/cb5d2f22-0dcc-457c-8931-f6f6fac769bd-kube-api-access-d8v8g\") pod \"cb5d2f22-0dcc-457c-8931-f6f6fac769bd\" (UID: \"cb5d2f22-0dcc-457c-8931-f6f6fac769bd\") "
Sep 10 23:15:37.833796 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.833651    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c68e804e-e181-401c-97a8-de5e91045f0b-plugins-config-volume\") pod \"c68e804e-e181-401c-97a8-de5e91045f0b\" (UID: \"c68e804e-e181-401c-97a8-de5e91045f0b\") "
Sep 10 23:15:37.833796 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.833692    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82shn\" (UniqueName: \"kubernetes.io/projected/c68e804e-e181-401c-97a8-de5e91045f0b-kube-api-access-82shn\") pod \"c68e804e-e181-401c-97a8-de5e91045f0b\" (UID: \"c68e804e-e181-401c-97a8-de5e91045f0b\") "
Sep 10 23:15:37.833896 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.833821    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5d2f22-0dcc-457c-8931-f6f6fac769bd-plugins-config-volume\") pod \"cb5d2f22-0dcc-457c-8931-f6f6fac769bd\" (UID: \"cb5d2f22-0dcc-457c-8931-f6f6fac769bd\") "
Sep 10 23:15:37.833993 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.833971    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68e804e-e181-401c-97a8-de5e91045f0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c68e804e-e181-401c-97a8-de5e91045f0b" (UID: "c68e804e-e181-401c-97a8-de5e91045f0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:15:37.834166 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.834145    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c68e804e-e181-401c-97a8-de5e91045f0b-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:15:37.834233 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.834155    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5d2f22-0dcc-457c-8931-f6f6fac769bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb5d2f22-0dcc-457c-8931-f6f6fac769bd" (UID: "cb5d2f22-0dcc-457c-8931-f6f6fac769bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:15:37.835717 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.835699    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5d2f22-0dcc-457c-8931-f6f6fac769bd-kube-api-access-d8v8g" (OuterVolumeSpecName: "kube-api-access-d8v8g") pod "cb5d2f22-0dcc-457c-8931-f6f6fac769bd" (UID: "cb5d2f22-0dcc-457c-8931-f6f6fac769bd"). InnerVolumeSpecName "kube-api-access-d8v8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:15:37.835759 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.835724    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68e804e-e181-401c-97a8-de5e91045f0b-kube-api-access-82shn" (OuterVolumeSpecName: "kube-api-access-82shn") pod "c68e804e-e181-401c-97a8-de5e91045f0b" (UID: "c68e804e-e181-401c-97a8-de5e91045f0b"). InnerVolumeSpecName "kube-api-access-82shn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:15:37.932177 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.932144    2563 generic.go:356] "Generic (PLEG): container finished" podID="c68e804e-e181-401c-97a8-de5e91045f0b" containerID="fbfc448d47cc9156624f885c0c559555a1f7b8ad2e2f33bc149db8264309c76a" exitCode=0
Sep 10 23:15:37.932322 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.932186    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"
Sep 10 23:15:37.932322 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.932231    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv" event={"ID":"c68e804e-e181-401c-97a8-de5e91045f0b","Type":"ContainerDied","Data":"fbfc448d47cc9156624f885c0c559555a1f7b8ad2e2f33bc149db8264309c76a"}
Sep 10 23:15:37.932322 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.932287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv" event={"ID":"c68e804e-e181-401c-97a8-de5e91045f0b","Type":"ContainerDied","Data":"0327b80960fa791ab67624af0bcd10c7a952f5bbfb81215ab3d8f5125b566022"}
Sep 10 23:15:37.932322 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.932316    2563 scope.go:117] "RemoveContainer" containerID="fbfc448d47cc9156624f885c0c559555a1f7b8ad2e2f33bc149db8264309c76a"
Sep 10 23:15:37.933727 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.933707    2563 generic.go:356] "Generic (PLEG): container finished" podID="cb5d2f22-0dcc-457c-8931-f6f6fac769bd" containerID="b1ef52d5d2951d2c8ecad86324d14dbfa9ff55860017a3463a4da31f39a70d96" exitCode=0
Sep 10 23:15:37.933839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.933742    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"
Sep 10 23:15:37.933839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.933734    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q" event={"ID":"cb5d2f22-0dcc-457c-8931-f6f6fac769bd","Type":"ContainerDied","Data":"b1ef52d5d2951d2c8ecad86324d14dbfa9ff55860017a3463a4da31f39a70d96"}
Sep 10 23:15:37.933923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.933879    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q" event={"ID":"cb5d2f22-0dcc-457c-8931-f6f6fac769bd","Type":"ContainerDied","Data":"d4e037cc37f4ff37db13b7f1ba941daa2bf21d43d80a77c523a4829eebf1a2d1"}
Sep 10 23:15:37.934807 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.934726    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8v8g\" (UniqueName: \"kubernetes.io/projected/cb5d2f22-0dcc-457c-8931-f6f6fac769bd-kube-api-access-d8v8g\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:15:37.934807 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.934750    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82shn\" (UniqueName: \"kubernetes.io/projected/c68e804e-e181-401c-97a8-de5e91045f0b-kube-api-access-82shn\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:15:37.934807 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.934766    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5d2f22-0dcc-457c-8931-f6f6fac769bd-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:15:37.942596 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.942573    2563 scope.go:117] "RemoveContainer" containerID="fbfc448d47cc9156624f885c0c559555a1f7b8ad2e2f33bc149db8264309c76a"
Sep 10 23:15:37.942925 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:15:37.942895    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbfc448d47cc9156624f885c0c559555a1f7b8ad2e2f33bc149db8264309c76a\": container with ID starting with fbfc448d47cc9156624f885c0c559555a1f7b8ad2e2f33bc149db8264309c76a not found: ID does not exist" containerID="fbfc448d47cc9156624f885c0c559555a1f7b8ad2e2f33bc149db8264309c76a"
Sep 10 23:15:37.942995 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.942932    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbfc448d47cc9156624f885c0c559555a1f7b8ad2e2f33bc149db8264309c76a"} err="failed to get container status \"fbfc448d47cc9156624f885c0c559555a1f7b8ad2e2f33bc149db8264309c76a\": rpc error: code = NotFound desc = could not find container \"fbfc448d47cc9156624f885c0c559555a1f7b8ad2e2f33bc149db8264309c76a\": container with ID starting with fbfc448d47cc9156624f885c0c559555a1f7b8ad2e2f33bc149db8264309c76a not found: ID does not exist"
Sep 10 23:15:37.942995 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.942954    2563 scope.go:117] "RemoveContainer" containerID="b1ef52d5d2951d2c8ecad86324d14dbfa9ff55860017a3463a4da31f39a70d96"
Sep 10 23:15:37.951090 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.950999    2563 scope.go:117] "RemoveContainer" containerID="b1ef52d5d2951d2c8ecad86324d14dbfa9ff55860017a3463a4da31f39a70d96"
Sep 10 23:15:37.951517 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:15:37.951472    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ef52d5d2951d2c8ecad86324d14dbfa9ff55860017a3463a4da31f39a70d96\": container with ID starting with b1ef52d5d2951d2c8ecad86324d14dbfa9ff55860017a3463a4da31f39a70d96 not found: ID does not exist" containerID="b1ef52d5d2951d2c8ecad86324d14dbfa9ff55860017a3463a4da31f39a70d96"
Sep 10 23:15:37.951662 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.951633    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ef52d5d2951d2c8ecad86324d14dbfa9ff55860017a3463a4da31f39a70d96"} err="failed to get container status \"b1ef52d5d2951d2c8ecad86324d14dbfa9ff55860017a3463a4da31f39a70d96\": rpc error: code = NotFound desc = could not find container \"b1ef52d5d2951d2c8ecad86324d14dbfa9ff55860017a3463a4da31f39a70d96\": container with ID starting with b1ef52d5d2951d2c8ecad86324d14dbfa9ff55860017a3463a4da31f39a70d96 not found: ID does not exist"
Sep 10 23:15:37.954182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.954161    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"]
Sep 10 23:15:37.956182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.956162    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7dbe617a-5cb86d978-zgzmv"]
Sep 10 23:15:37.967238 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.967199    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"]
Sep 10 23:15:37.969012 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:37.968994    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7dbe617a-6f5598bb94-5sz4q"]
Sep 10 23:15:39.814698 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:39.814656    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c68e804e-e181-401c-97a8-de5e91045f0b" path="/var/lib/kubelet/pods/c68e804e-e181-401c-97a8-de5e91045f0b/volumes"
Sep 10 23:15:39.815195 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:39.815027    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb5d2f22-0dcc-457c-8931-f6f6fac769bd" path="/var/lib/kubelet/pods/cb5d2f22-0dcc-457c-8931-f6f6fac769bd/volumes"
Sep 10 23:15:42.953781 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.953745    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"]
Sep 10 23:15:42.954198 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.954183    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c68e804e-e181-401c-97a8-de5e91045f0b" containerName="payload-pre-processing"
Sep 10 23:15:42.954243 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.954199    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c68e804e-e181-401c-97a8-de5e91045f0b" containerName="payload-pre-processing"
Sep 10 23:15:42.954243 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.954212    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abf0d31b-3556-4c79-9f24-e3f3112e50c0" containerName="revoke-keys"
Sep 10 23:15:42.954243 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.954218    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="abf0d31b-3556-4c79-9f24-e3f3112e50c0" containerName="revoke-keys"
Sep 10 23:15:42.954243 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.954229    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5d2f22-0dcc-457c-8931-f6f6fac769bd" containerName="payload-processing"
Sep 10 23:15:42.954243 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.954235    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5d2f22-0dcc-457c-8931-f6f6fac769bd" containerName="payload-processing"
Sep 10 23:15:42.954400 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.954297    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5d2f22-0dcc-457c-8931-f6f6fac769bd" containerName="payload-processing"
Sep 10 23:15:42.954400 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.954312    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="abf0d31b-3556-4c79-9f24-e3f3112e50c0" containerName="revoke-keys"
Sep 10 23:15:42.954400 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.954322    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c68e804e-e181-401c-97a8-de5e91045f0b" containerName="payload-pre-processing"
Sep 10 23:15:42.958854 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.958838    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"
Sep 10 23:15:42.961213 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.961179    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c5c3fdb6\""
Sep 10 23:15:42.961341 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.961287    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c5c3fdb6-dockercfg-bqplh\""
Sep 10 23:15:42.968823 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.968795    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"]
Sep 10 23:15:42.981969 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.981938    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"]
Sep 10 23:15:42.985414 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.985397    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"
Sep 10 23:15:42.995600 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:42.995574    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"]
Sep 10 23:15:43.089387 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.089350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m7tl\" (UniqueName: \"kubernetes.io/projected/c0105b19-8b3b-464a-b86d-98699dae7667-kube-api-access-6m7tl\") pod \"payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5\" (UID: \"c0105b19-8b3b-464a-b86d-98699dae7667\") " pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"
Sep 10 23:15:43.089387 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.089388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0105b19-8b3b-464a-b86d-98699dae7667-plugins-config-volume\") pod \"payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5\" (UID: \"c0105b19-8b3b-464a-b86d-98699dae7667\") " pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"
Sep 10 23:15:43.089668 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.089418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xk2t\" (UniqueName: \"kubernetes.io/projected/99f0715c-f0c9-4220-8e7b-7c267b2bdce8-kube-api-access-7xk2t\") pod \"payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl\" (UID: \"99f0715c-f0c9-4220-8e7b-7c267b2bdce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"
Sep 10 23:15:43.089668 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.089532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f0715c-f0c9-4220-8e7b-7c267b2bdce8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl\" (UID: \"99f0715c-f0c9-4220-8e7b-7c267b2bdce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"
Sep 10 23:15:43.190361 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.190326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m7tl\" (UniqueName: \"kubernetes.io/projected/c0105b19-8b3b-464a-b86d-98699dae7667-kube-api-access-6m7tl\") pod \"payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5\" (UID: \"c0105b19-8b3b-464a-b86d-98699dae7667\") " pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"
Sep 10 23:15:43.190361 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.190363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0105b19-8b3b-464a-b86d-98699dae7667-plugins-config-volume\") pod \"payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5\" (UID: \"c0105b19-8b3b-464a-b86d-98699dae7667\") " pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"
Sep 10 23:15:43.190631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.190400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xk2t\" (UniqueName: \"kubernetes.io/projected/99f0715c-f0c9-4220-8e7b-7c267b2bdce8-kube-api-access-7xk2t\") pod \"payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl\" (UID: \"99f0715c-f0c9-4220-8e7b-7c267b2bdce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"
Sep 10 23:15:43.190631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.190429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f0715c-f0c9-4220-8e7b-7c267b2bdce8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl\" (UID: \"99f0715c-f0c9-4220-8e7b-7c267b2bdce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"
Sep 10 23:15:43.191118 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.191097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f0715c-f0c9-4220-8e7b-7c267b2bdce8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl\" (UID: \"99f0715c-f0c9-4220-8e7b-7c267b2bdce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"
Sep 10 23:15:43.191118 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.191110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0105b19-8b3b-464a-b86d-98699dae7667-plugins-config-volume\") pod \"payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5\" (UID: \"c0105b19-8b3b-464a-b86d-98699dae7667\") " pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"
Sep 10 23:15:43.198827 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.198799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xk2t\" (UniqueName: \"kubernetes.io/projected/99f0715c-f0c9-4220-8e7b-7c267b2bdce8-kube-api-access-7xk2t\") pod \"payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl\" (UID: \"99f0715c-f0c9-4220-8e7b-7c267b2bdce8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"
Sep 10 23:15:43.198943 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.198903    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m7tl\" (UniqueName: \"kubernetes.io/projected/c0105b19-8b3b-464a-b86d-98699dae7667-kube-api-access-6m7tl\") pod \"payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5\" (UID: \"c0105b19-8b3b-464a-b86d-98699dae7667\") " pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"
Sep 10 23:15:43.270773 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.270690    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"
Sep 10 23:15:43.296974 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.296943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"
Sep 10 23:15:43.546083 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.546054    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"]
Sep 10 23:15:43.547942 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:15:43.547913    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f0715c_f0c9_4220_8e7b_7c267b2bdce8.slice/crio-4abd8ac862c9d00c7709390f38768f88f8dddbd88337ba1ab6bd487a6c899784 WatchSource:0}: Error finding container 4abd8ac862c9d00c7709390f38768f88f8dddbd88337ba1ab6bd487a6c899784: Status 404 returned error can't find the container with id 4abd8ac862c9d00c7709390f38768f88f8dddbd88337ba1ab6bd487a6c899784
Sep 10 23:15:43.577104 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.577080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"]
Sep 10 23:15:43.593665 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:15:43.593637    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0105b19_8b3b_464a_b86d_98699dae7667.slice/crio-92438060ac221d9a9f61a5870e884e317b96f12e55f0ef2ec78543162a0f5412 WatchSource:0}: Error finding container 92438060ac221d9a9f61a5870e884e317b96f12e55f0ef2ec78543162a0f5412: Status 404 returned error can't find the container with id 92438060ac221d9a9f61a5870e884e317b96f12e55f0ef2ec78543162a0f5412
Sep 10 23:15:43.979717 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.979680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5" event={"ID":"c0105b19-8b3b-464a-b86d-98699dae7667","Type":"ContainerStarted","Data":"e0d5d7723b57350b33a1a20a03bbb9f89e9320f84163d8d2a255a79ac801c37f"}
Sep 10 23:15:43.980205 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.979748    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5" event={"ID":"c0105b19-8b3b-464a-b86d-98699dae7667","Type":"ContainerStarted","Data":"92438060ac221d9a9f61a5870e884e317b96f12e55f0ef2ec78543162a0f5412"}
Sep 10 23:15:43.980205 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.979785    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"
Sep 10 23:15:43.981445 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.981417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl" event={"ID":"99f0715c-f0c9-4220-8e7b-7c267b2bdce8","Type":"ContainerStarted","Data":"e5f0e6d3b10c5113981eaebc32f55749503aad872e289aff37f5459fdc79320b"}
Sep 10 23:15:43.981593 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.981462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl" event={"ID":"99f0715c-f0c9-4220-8e7b-7c267b2bdce8","Type":"ContainerStarted","Data":"4abd8ac862c9d00c7709390f38768f88f8dddbd88337ba1ab6bd487a6c899784"}
Sep 10 23:15:43.981593 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.981507    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"
Sep 10 23:15:43.996430 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:43.996381    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5" podStartSLOduration=1.9963632439999999 podStartE2EDuration="1.996363244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:43.995408803 +0000 UTC m=+864.686986019" watchObservedRunningTime="2026-09-10 23:15:43.996363244 +0000 UTC m=+864.687940433"
Sep 10 23:15:44.010572 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:44.009883    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl" podStartSLOduration=2.009863315 podStartE2EDuration="2.009863315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:44.008875827 +0000 UTC m=+864.700453034" watchObservedRunningTime="2026-09-10 23:15:44.009863315 +0000 UTC m=+864.701440501"
Sep 10 23:15:54.988874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:54.988841    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"
Sep 10 23:15:54.989381 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:15:54.989022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"
Sep 10 23:16:02.351615 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.351582    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"]
Sep 10 23:16:02.355414 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.355396    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.358634 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.358609    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-689fe3\"/\"autores-model-689fe3-kserve-self-signed-certs\""
Sep 10 23:16:02.358744 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.358609    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-689fe3\"/\"openshift-service-ca.crt\""
Sep 10 23:16:02.358744 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.358645    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-689fe3\"/\"kube-root-ca.crt\""
Sep 10 23:16:02.358744 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.358610    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-689fe3\"/\"default-dockercfg-x2khg\""
Sep 10 23:16:02.364462 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.364439    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"]
Sep 10 23:16:02.367666 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.367646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-home\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.367773 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.367692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-dshm\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.367773 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.367738    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlncr\" (UniqueName: \"kubernetes.io/projected/45ffa2b2-1ad9-4384-8c83-e37076824c49-kube-api-access-hlncr\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.367852 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.367773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-tmp-dir\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.367852 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.367832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45ffa2b2-1ad9-4384-8c83-e37076824c49-tls-certs\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.367937 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.367921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-model-cache\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.469042 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.469005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-home\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.469231 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.469067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-dshm\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.469231 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.469098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlncr\" (UniqueName: \"kubernetes.io/projected/45ffa2b2-1ad9-4384-8c83-e37076824c49-kube-api-access-hlncr\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.469231 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.469132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-tmp-dir\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.469231 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.469188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45ffa2b2-1ad9-4384-8c83-e37076824c49-tls-certs\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.469437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.469247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-model-cache\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.469437 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.469361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-home\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.469599 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.469537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-tmp-dir\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.469661 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.469637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-model-cache\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.471212 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.471193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-dshm\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.471573 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.471557    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45ffa2b2-1ad9-4384-8c83-e37076824c49-tls-certs\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.476640 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.476622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlncr\" (UniqueName: \"kubernetes.io/projected/45ffa2b2-1ad9-4384-8c83-e37076824c49-kube-api-access-hlncr\") pod \"autores-model-689fe3-kserve-d5b47cb6c-p7mbv\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") " pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.666606 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.666516    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:02.942454 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:02.942426    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"]
Sep 10 23:16:02.944509 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:16:02.944463    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45ffa2b2_1ad9_4384_8c83_e37076824c49.slice/crio-fad09bf73d83e74757a613164d07595c38085251b62048deeaff7ac6e9b88ad1 WatchSource:0}: Error finding container fad09bf73d83e74757a613164d07595c38085251b62048deeaff7ac6e9b88ad1: Status 404 returned error can't find the container with id fad09bf73d83e74757a613164d07595c38085251b62048deeaff7ac6e9b88ad1
Sep 10 23:16:03.082793 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:03.082763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv" event={"ID":"45ffa2b2-1ad9-4384-8c83-e37076824c49","Type":"ContainerStarted","Data":"cfc260f9bde2fa2d7d340eba5cd5d59d44a5486b22f1c7d1cadf147a5027a6b1"}
Sep 10 23:16:03.082971 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:03.082801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv" event={"ID":"45ffa2b2-1ad9-4384-8c83-e37076824c49","Type":"ContainerStarted","Data":"fad09bf73d83e74757a613164d07595c38085251b62048deeaff7ac6e9b88ad1"}
Sep 10 23:16:03.100068 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:03.100014    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv" podStartSLOduration=1.099997714 podStartE2EDuration="1.099997714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:16:03.097914862 +0000 UTC m=+883.789492067" watchObservedRunningTime="2026-09-10 23:16:03.099997714 +0000 UTC m=+883.791574934"
Sep 10 23:16:06.731386 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:06.731351    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf"]
Sep 10 23:16:06.735098 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:06.735082    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf"
Sep 10 23:16:06.737838 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:06.737815    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:16:06.737941 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:06.737846    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:16:06.742264 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:06.742243    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf"]
Sep 10 23:16:06.812019 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:06.811985    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651deba5-20d7-4fcf-8a52-657a1b8cf258-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf\" (UID: \"651deba5-20d7-4fcf-8a52-657a1b8cf258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf"
Sep 10 23:16:06.912536 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:06.912506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651deba5-20d7-4fcf-8a52-657a1b8cf258-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf\" (UID: \"651deba5-20d7-4fcf-8a52-657a1b8cf258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf"
Sep 10 23:16:06.913085 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:06.913066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651deba5-20d7-4fcf-8a52-657a1b8cf258-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf\" (UID: \"651deba5-20d7-4fcf-8a52-657a1b8cf258\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf"
Sep 10 23:16:07.075850 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:07.075821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf"
Sep 10 23:16:07.345988 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:07.345963    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf"]
Sep 10 23:16:07.347684 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:16:07.347652    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651deba5_20d7_4fcf_8a52_657a1b8cf258.slice/crio-eda0d64bfffe0ce717e9f2ee465efb3e73ccc477142fe423857071fdcf5a62d6 WatchSource:0}: Error finding container eda0d64bfffe0ce717e9f2ee465efb3e73ccc477142fe423857071fdcf5a62d6: Status 404 returned error can't find the container with id eda0d64bfffe0ce717e9f2ee465efb3e73ccc477142fe423857071fdcf5a62d6
Sep 10 23:16:08.111235 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:08.111202    2563 generic.go:356] "Generic (PLEG): container finished" podID="651deba5-20d7-4fcf-8a52-657a1b8cf258" containerID="434d5599a18f3a9a4a5efe9fca95dcf35f8f5369912096a83a677b5bc0e11b8b" exitCode=0
Sep 10 23:16:08.111642 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:08.111263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf" event={"ID":"651deba5-20d7-4fcf-8a52-657a1b8cf258","Type":"ContainerDied","Data":"434d5599a18f3a9a4a5efe9fca95dcf35f8f5369912096a83a677b5bc0e11b8b"}
Sep 10 23:16:08.111642 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:08.111329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf" event={"ID":"651deba5-20d7-4fcf-8a52-657a1b8cf258","Type":"ContainerStarted","Data":"eda0d64bfffe0ce717e9f2ee465efb3e73ccc477142fe423857071fdcf5a62d6"}
Sep 10 23:16:09.252016 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:09.251991    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf"
Sep 10 23:16:09.335739 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:09.335709    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651deba5-20d7-4fcf-8a52-657a1b8cf258-maas-api-service-ca\") pod \"651deba5-20d7-4fcf-8a52-657a1b8cf258\" (UID: \"651deba5-20d7-4fcf-8a52-657a1b8cf258\") "
Sep 10 23:16:09.336062 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:09.336040    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651deba5-20d7-4fcf-8a52-657a1b8cf258-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "651deba5-20d7-4fcf-8a52-657a1b8cf258" (UID: "651deba5-20d7-4fcf-8a52-657a1b8cf258"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:09.436414 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:09.436339    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651deba5-20d7-4fcf-8a52-657a1b8cf258-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:10.123232 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:10.123148    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf"
Sep 10 23:16:10.123232 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:10.123181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf" event={"ID":"651deba5-20d7-4fcf-8a52-657a1b8cf258","Type":"ContainerDied","Data":"eda0d64bfffe0ce717e9f2ee465efb3e73ccc477142fe423857071fdcf5a62d6"}
Sep 10 23:16:10.123414 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:10.123250    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eda0d64bfffe0ce717e9f2ee465efb3e73ccc477142fe423857071fdcf5a62d6"
Sep 10 23:16:12.667743 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:12.667664    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:12.667743 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:12.667715    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:12.680225 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:12.680204    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:12.894334 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:12.894298    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"]
Sep 10 23:16:12.894619 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:12.894590    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5" podUID="c0105b19-8b3b-464a-b86d-98699dae7667" containerName="payload-processing" containerID="cri-o://e0d5d7723b57350b33a1a20a03bbb9f89e9320f84163d8d2a255a79ac801c37f" gracePeriod=30
Sep 10 23:16:12.903429 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:12.902576    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"]
Sep 10 23:16:12.903429 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:12.902778    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl" podUID="99f0715c-f0c9-4220-8e7b-7c267b2bdce8" containerName="payload-pre-processing" containerID="cri-o://e5f0e6d3b10c5113981eaebc32f55749503aad872e289aff37f5459fdc79320b" gracePeriod=30
Sep 10 23:16:13.057847 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.057811    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf"]
Sep 10 23:16:13.059939 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.059916    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c5c3fdb6-65dnf"]
Sep 10 23:16:13.143168 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.143136    2563 generic.go:356] "Generic (PLEG): container finished" podID="99f0715c-f0c9-4220-8e7b-7c267b2bdce8" containerID="e5f0e6d3b10c5113981eaebc32f55749503aad872e289aff37f5459fdc79320b" exitCode=0
Sep 10 23:16:13.143330 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.143178    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl" event={"ID":"99f0715c-f0c9-4220-8e7b-7c267b2bdce8","Type":"ContainerDied","Data":"e5f0e6d3b10c5113981eaebc32f55749503aad872e289aff37f5459fdc79320b"}
Sep 10 23:16:13.145033 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.145012    2563 generic.go:356] "Generic (PLEG): container finished" podID="c0105b19-8b3b-464a-b86d-98699dae7667" containerID="e0d5d7723b57350b33a1a20a03bbb9f89e9320f84163d8d2a255a79ac801c37f" exitCode=0
Sep 10 23:16:13.145183 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.145085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5" event={"ID":"c0105b19-8b3b-464a-b86d-98699dae7667","Type":"ContainerDied","Data":"e0d5d7723b57350b33a1a20a03bbb9f89e9320f84163d8d2a255a79ac801c37f"}
Sep 10 23:16:13.156960 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.156938    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:13.167862 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.167846    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"
Sep 10 23:16:13.183467 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.183402    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"
Sep 10 23:16:13.274056 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.274022    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0105b19-8b3b-464a-b86d-98699dae7667-plugins-config-volume\") pod \"c0105b19-8b3b-464a-b86d-98699dae7667\" (UID: \"c0105b19-8b3b-464a-b86d-98699dae7667\") "
Sep 10 23:16:13.274200 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.274096    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m7tl\" (UniqueName: \"kubernetes.io/projected/c0105b19-8b3b-464a-b86d-98699dae7667-kube-api-access-6m7tl\") pod \"c0105b19-8b3b-464a-b86d-98699dae7667\" (UID: \"c0105b19-8b3b-464a-b86d-98699dae7667\") "
Sep 10 23:16:13.274200 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.274157    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xk2t\" (UniqueName: \"kubernetes.io/projected/99f0715c-f0c9-4220-8e7b-7c267b2bdce8-kube-api-access-7xk2t\") pod \"99f0715c-f0c9-4220-8e7b-7c267b2bdce8\" (UID: \"99f0715c-f0c9-4220-8e7b-7c267b2bdce8\") "
Sep 10 23:16:13.274200 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.274194    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f0715c-f0c9-4220-8e7b-7c267b2bdce8-plugins-config-volume\") pod \"99f0715c-f0c9-4220-8e7b-7c267b2bdce8\" (UID: \"99f0715c-f0c9-4220-8e7b-7c267b2bdce8\") "
Sep 10 23:16:13.274484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.274444    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0105b19-8b3b-464a-b86d-98699dae7667-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0105b19-8b3b-464a-b86d-98699dae7667" (UID: "c0105b19-8b3b-464a-b86d-98699dae7667"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:13.274694 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.274670    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f0715c-f0c9-4220-8e7b-7c267b2bdce8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99f0715c-f0c9-4220-8e7b-7c267b2bdce8" (UID: "99f0715c-f0c9-4220-8e7b-7c267b2bdce8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:13.276148 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.276131    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0105b19-8b3b-464a-b86d-98699dae7667-kube-api-access-6m7tl" (OuterVolumeSpecName: "kube-api-access-6m7tl") pod "c0105b19-8b3b-464a-b86d-98699dae7667" (UID: "c0105b19-8b3b-464a-b86d-98699dae7667"). InnerVolumeSpecName "kube-api-access-6m7tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:13.276199 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.276182    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f0715c-f0c9-4220-8e7b-7c267b2bdce8-kube-api-access-7xk2t" (OuterVolumeSpecName: "kube-api-access-7xk2t") pod "99f0715c-f0c9-4220-8e7b-7c267b2bdce8" (UID: "99f0715c-f0c9-4220-8e7b-7c267b2bdce8"). InnerVolumeSpecName "kube-api-access-7xk2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:13.375518 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.375467    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0105b19-8b3b-464a-b86d-98699dae7667-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:13.375518 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.375517    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m7tl\" (UniqueName: \"kubernetes.io/projected/c0105b19-8b3b-464a-b86d-98699dae7667-kube-api-access-6m7tl\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:13.375716 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.375529    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xk2t\" (UniqueName: \"kubernetes.io/projected/99f0715c-f0c9-4220-8e7b-7c267b2bdce8-kube-api-access-7xk2t\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:13.375716 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.375537    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f0715c-f0c9-4220-8e7b-7c267b2bdce8-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:13.814533 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:13.814471    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="651deba5-20d7-4fcf-8a52-657a1b8cf258" path="/var/lib/kubelet/pods/651deba5-20d7-4fcf-8a52-657a1b8cf258/volumes"
Sep 10 23:16:14.152484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:14.152385    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"
Sep 10 23:16:14.152484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:14.152405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5" event={"ID":"c0105b19-8b3b-464a-b86d-98699dae7667","Type":"ContainerDied","Data":"92438060ac221d9a9f61a5870e884e317b96f12e55f0ef2ec78543162a0f5412"}
Sep 10 23:16:14.152716 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:14.152531    2563 scope.go:117] "RemoveContainer" containerID="e0d5d7723b57350b33a1a20a03bbb9f89e9320f84163d8d2a255a79ac801c37f"
Sep 10 23:16:14.154303 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:14.154274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl" event={"ID":"99f0715c-f0c9-4220-8e7b-7c267b2bdce8","Type":"ContainerDied","Data":"4abd8ac862c9d00c7709390f38768f88f8dddbd88337ba1ab6bd487a6c899784"}
Sep 10 23:16:14.154456 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:14.154421    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"
Sep 10 23:16:14.162435 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:14.162419    2563 scope.go:117] "RemoveContainer" containerID="e5f0e6d3b10c5113981eaebc32f55749503aad872e289aff37f5459fdc79320b"
Sep 10 23:16:14.172690 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:14.172636    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"]
Sep 10 23:16:14.176064 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:14.176040    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c5c3fdb6-5bc86d7df4-fksxl"]
Sep 10 23:16:14.187056 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:14.187031    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"]
Sep 10 23:16:14.190622 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:14.190600    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c5c3fdb6-5c8489fd4c-zwwl5"]
Sep 10 23:16:15.814623 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:15.814590    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99f0715c-f0c9-4220-8e7b-7c267b2bdce8" path="/var/lib/kubelet/pods/99f0715c-f0c9-4220-8e7b-7c267b2bdce8/volumes"
Sep 10 23:16:15.815832 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:15.815816    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0105b19-8b3b-464a-b86d-98699dae7667" path="/var/lib/kubelet/pods/c0105b19-8b3b-464a-b86d-98699dae7667/volumes"
Sep 10 23:16:19.178169 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.178135    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"]
Sep 10 23:16:19.178708 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.178659    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="651deba5-20d7-4fcf-8a52-657a1b8cf258" containerName="revoke-keys"
Sep 10 23:16:19.178708 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.178676    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="651deba5-20d7-4fcf-8a52-657a1b8cf258" containerName="revoke-keys"
Sep 10 23:16:19.178708 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.178696    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0105b19-8b3b-464a-b86d-98699dae7667" containerName="payload-processing"
Sep 10 23:16:19.178708 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.178702    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0105b19-8b3b-464a-b86d-98699dae7667" containerName="payload-processing"
Sep 10 23:16:19.178708 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.178712    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99f0715c-f0c9-4220-8e7b-7c267b2bdce8" containerName="payload-pre-processing"
Sep 10 23:16:19.178984 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.178718    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="99f0715c-f0c9-4220-8e7b-7c267b2bdce8" containerName="payload-pre-processing"
Sep 10 23:16:19.178984 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.178788    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="651deba5-20d7-4fcf-8a52-657a1b8cf258" containerName="revoke-keys"
Sep 10 23:16:19.178984 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.178797    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0105b19-8b3b-464a-b86d-98699dae7667" containerName="payload-processing"
Sep 10 23:16:19.178984 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.178805    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="99f0715c-f0c9-4220-8e7b-7c267b2bdce8" containerName="payload-pre-processing"
Sep 10 23:16:19.182162 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.182141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"
Sep 10 23:16:19.185472 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.185432    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-51383f0b-dockercfg-pwnrc\""
Sep 10 23:16:19.186031 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.185773    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-51383f0b\""
Sep 10 23:16:19.194361 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.194277    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"]
Sep 10 23:16:19.335086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.335052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0b3b13-5c79-47f1-a548-88da251fd1cf-plugins-config-volume\") pod \"payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z\" (UID: \"dc0b3b13-5c79-47f1-a548-88da251fd1cf\") " pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"
Sep 10 23:16:19.335251 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.335106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwswd\" (UniqueName: \"kubernetes.io/projected/dc0b3b13-5c79-47f1-a548-88da251fd1cf-kube-api-access-bwswd\") pod \"payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z\" (UID: \"dc0b3b13-5c79-47f1-a548-88da251fd1cf\") " pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"
Sep 10 23:16:19.436696 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.436617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwswd\" (UniqueName: \"kubernetes.io/projected/dc0b3b13-5c79-47f1-a548-88da251fd1cf-kube-api-access-bwswd\") pod \"payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z\" (UID: \"dc0b3b13-5c79-47f1-a548-88da251fd1cf\") " pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"
Sep 10 23:16:19.436839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.436729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0b3b13-5c79-47f1-a548-88da251fd1cf-plugins-config-volume\") pod \"payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z\" (UID: \"dc0b3b13-5c79-47f1-a548-88da251fd1cf\") " pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"
Sep 10 23:16:19.437363 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.437344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0b3b13-5c79-47f1-a548-88da251fd1cf-plugins-config-volume\") pod \"payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z\" (UID: \"dc0b3b13-5c79-47f1-a548-88da251fd1cf\") " pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"
Sep 10 23:16:19.444461 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.444438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwswd\" (UniqueName: \"kubernetes.io/projected/dc0b3b13-5c79-47f1-a548-88da251fd1cf-kube-api-access-bwswd\") pod \"payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z\" (UID: \"dc0b3b13-5c79-47f1-a548-88da251fd1cf\") " pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"
Sep 10 23:16:19.498200 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.498173    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"
Sep 10 23:16:19.769430 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.769405    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"]
Sep 10 23:16:19.771601 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:16:19.771574    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0b3b13_5c79_47f1_a548_88da251fd1cf.slice/crio-30f2997b97b4dec0fcb804e2080c4d11bbed841143206c71557793e439c24ab6 WatchSource:0}: Error finding container 30f2997b97b4dec0fcb804e2080c4d11bbed841143206c71557793e439c24ab6: Status 404 returned error can't find the container with id 30f2997b97b4dec0fcb804e2080c4d11bbed841143206c71557793e439c24ab6
Sep 10 23:16:19.814517 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.814468    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:16:19.814676 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:19.814573    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:16:20.193788 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:20.193754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z" event={"ID":"dc0b3b13-5c79-47f1-a548-88da251fd1cf","Type":"ContainerStarted","Data":"f92162cf806bc93d33037551f5ee5b87ba2bad7648aaa237926fcb1ef0cf5ff0"}
Sep 10 23:16:20.194164 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:20.193812    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z" event={"ID":"dc0b3b13-5c79-47f1-a548-88da251fd1cf","Type":"ContainerStarted","Data":"30f2997b97b4dec0fcb804e2080c4d11bbed841143206c71557793e439c24ab6"}
Sep 10 23:16:20.194164 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:20.193849    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"
Sep 10 23:16:20.212382 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:20.212329    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z" podStartSLOduration=1.212313677 podStartE2EDuration="1.212313677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:16:20.210040478 +0000 UTC m=+900.901617709" watchObservedRunningTime="2026-09-10 23:16:20.212313677 +0000 UTC m=+900.903890860"
Sep 10 23:16:31.201718 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:31.201673    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"
Sep 10 23:16:37.174044 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:37.174005    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7"]
Sep 10 23:16:37.177763 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:37.177744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7"
Sep 10 23:16:37.180218 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:37.180195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:16:37.180319 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:37.180259    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:16:37.186207 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:37.186180    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7"]
Sep 10 23:16:37.203074 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:37.203041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89f7a2e-7339-4fa3-803c-be7d4ed348fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-689fe3-s7sh7\" (UID: \"e89f7a2e-7339-4fa3-803c-be7d4ed348fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7"
Sep 10 23:16:37.304154 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:37.304124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89f7a2e-7339-4fa3-803c-be7d4ed348fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-689fe3-s7sh7\" (UID: \"e89f7a2e-7339-4fa3-803c-be7d4ed348fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7"
Sep 10 23:16:37.304793 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:37.304774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89f7a2e-7339-4fa3-803c-be7d4ed348fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-689fe3-s7sh7\" (UID: \"e89f7a2e-7339-4fa3-803c-be7d4ed348fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7"
Sep 10 23:16:37.489993 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:37.489897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7"
Sep 10 23:16:37.762899 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:37.762810    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7"]
Sep 10 23:16:37.765537 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:16:37.765480    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89f7a2e_7339_4fa3_803c_be7d4ed348fd.slice/crio-b3e097e11dfe54fcd77533ea3734456f16e3bce62766f96d879d77b189b4acdb WatchSource:0}: Error finding container b3e097e11dfe54fcd77533ea3734456f16e3bce62766f96d879d77b189b4acdb: Status 404 returned error can't find the container with id b3e097e11dfe54fcd77533ea3734456f16e3bce62766f96d879d77b189b4acdb
Sep 10 23:16:38.305938 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:38.305852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7" event={"ID":"e89f7a2e-7339-4fa3-803c-be7d4ed348fd","Type":"ContainerStarted","Data":"6bfa3585d5d4290eb9045f5e9d2c1ecf7b4e72febf25a79186270bfa03fe991d"}
Sep 10 23:16:38.305938 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:38.305906    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7" event={"ID":"e89f7a2e-7339-4fa3-803c-be7d4ed348fd","Type":"ContainerStarted","Data":"b3e097e11dfe54fcd77533ea3734456f16e3bce62766f96d879d77b189b4acdb"}
Sep 10 23:16:38.321475 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:38.321425    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7" podStartSLOduration=1.321410272 podStartE2EDuration="1.321410272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:16:38.320540695 +0000 UTC m=+919.012117910" watchObservedRunningTime="2026-09-10 23:16:38.321410272 +0000 UTC m=+919.012987456"
Sep 10 23:16:39.312381 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:39.312345    2563 generic.go:356] "Generic (PLEG): container finished" podID="e89f7a2e-7339-4fa3-803c-be7d4ed348fd" containerID="6bfa3585d5d4290eb9045f5e9d2c1ecf7b4e72febf25a79186270bfa03fe991d" exitCode=0
Sep 10 23:16:39.312794 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:39.312431    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7" event={"ID":"e89f7a2e-7339-4fa3-803c-be7d4ed348fd","Type":"ContainerDied","Data":"6bfa3585d5d4290eb9045f5e9d2c1ecf7b4e72febf25a79186270bfa03fe991d"}
Sep 10 23:16:40.471333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:40.471310    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7"
Sep 10 23:16:40.540894 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:40.540849    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89f7a2e-7339-4fa3-803c-be7d4ed348fd-maas-api-service-ca\") pod \"e89f7a2e-7339-4fa3-803c-be7d4ed348fd\" (UID: \"e89f7a2e-7339-4fa3-803c-be7d4ed348fd\") "
Sep 10 23:16:40.541213 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:40.541189    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89f7a2e-7339-4fa3-803c-be7d4ed348fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e89f7a2e-7339-4fa3-803c-be7d4ed348fd" (UID: "e89f7a2e-7339-4fa3-803c-be7d4ed348fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:40.642331 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:40.642242    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89f7a2e-7339-4fa3-803c-be7d4ed348fd-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:41.324903 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:41.324865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7" event={"ID":"e89f7a2e-7339-4fa3-803c-be7d4ed348fd","Type":"ContainerDied","Data":"b3e097e11dfe54fcd77533ea3734456f16e3bce62766f96d879d77b189b4acdb"}
Sep 10 23:16:41.324903 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:41.324888    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7"
Sep 10 23:16:41.325134 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:41.324919    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3e097e11dfe54fcd77533ea3734456f16e3bce62766f96d879d77b189b4acdb"
Sep 10 23:16:42.066966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:42.066936    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h"]
Sep 10 23:16:42.067378 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:42.067365    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e89f7a2e-7339-4fa3-803c-be7d4ed348fd" containerName="revoke-keys"
Sep 10 23:16:42.067419 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:42.067379    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e89f7a2e-7339-4fa3-803c-be7d4ed348fd" containerName="revoke-keys"
Sep 10 23:16:42.067463 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:42.067454    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e89f7a2e-7339-4fa3-803c-be7d4ed348fd" containerName="revoke-keys"
Sep 10 23:16:42.070815 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:42.070795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h"
Sep 10 23:16:42.072821 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:42.072802    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:16:42.072943 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:42.072926    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:16:42.076827 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:42.076799    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h"]
Sep 10 23:16:42.156715 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:42.156682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc8b40-59aa-4695-a427-09d9b98a698e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h\" (UID: \"25cc8b40-59aa-4695-a427-09d9b98a698e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h"
Sep 10 23:16:42.257259 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:42.257219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc8b40-59aa-4695-a427-09d9b98a698e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h\" (UID: \"25cc8b40-59aa-4695-a427-09d9b98a698e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h"
Sep 10 23:16:42.257854 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:42.257836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc8b40-59aa-4695-a427-09d9b98a698e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h\" (UID: \"25cc8b40-59aa-4695-a427-09d9b98a698e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h"
Sep 10 23:16:42.383850 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:42.383759    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h"
Sep 10 23:16:42.653784 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:42.653759    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h"]
Sep 10 23:16:42.655865 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:16:42.655839    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cc8b40_59aa_4695_a427_09d9b98a698e.slice/crio-d931b646c434af9ff2229f55fe67429e79e40bd402ecbf6fd2d962b08f9b3938 WatchSource:0}: Error finding container d931b646c434af9ff2229f55fe67429e79e40bd402ecbf6fd2d962b08f9b3938: Status 404 returned error can't find the container with id d931b646c434af9ff2229f55fe67429e79e40bd402ecbf6fd2d962b08f9b3938
Sep 10 23:16:43.337336 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:43.337302    2563 generic.go:356] "Generic (PLEG): container finished" podID="25cc8b40-59aa-4695-a427-09d9b98a698e" containerID="5e6f7d4ed8c5de2159433d34384f5a6549a3a686f9eee2397dddeb9d76fa3a09" exitCode=0
Sep 10 23:16:43.337763 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:43.337373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h" event={"ID":"25cc8b40-59aa-4695-a427-09d9b98a698e","Type":"ContainerDied","Data":"5e6f7d4ed8c5de2159433d34384f5a6549a3a686f9eee2397dddeb9d76fa3a09"}
Sep 10 23:16:43.337763 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:43.337408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h" event={"ID":"25cc8b40-59aa-4695-a427-09d9b98a698e","Type":"ContainerStarted","Data":"d931b646c434af9ff2229f55fe67429e79e40bd402ecbf6fd2d962b08f9b3938"}
Sep 10 23:16:43.438479 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:43.438441    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"]
Sep 10 23:16:44.473440 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:44.473407    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h"
Sep 10 23:16:44.586214 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:44.586155    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc8b40-59aa-4695-a427-09d9b98a698e-maas-api-service-ca\") pod \"25cc8b40-59aa-4695-a427-09d9b98a698e\" (UID: \"25cc8b40-59aa-4695-a427-09d9b98a698e\") "
Sep 10 23:16:44.586529 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:44.586485    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25cc8b40-59aa-4695-a427-09d9b98a698e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25cc8b40-59aa-4695-a427-09d9b98a698e" (UID: "25cc8b40-59aa-4695-a427-09d9b98a698e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:44.687882 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:44.687801    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25cc8b40-59aa-4695-a427-09d9b98a698e-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:45.349101 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:45.349067    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h"
Sep 10 23:16:45.349274 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:45.349061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h" event={"ID":"25cc8b40-59aa-4695-a427-09d9b98a698e","Type":"ContainerDied","Data":"d931b646c434af9ff2229f55fe67429e79e40bd402ecbf6fd2d962b08f9b3938"}
Sep 10 23:16:45.349274 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:45.349207    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d931b646c434af9ff2229f55fe67429e79e40bd402ecbf6fd2d962b08f9b3938"
Sep 10 23:16:47.867394 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:47.867359    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-689fe3-29818035-tcg82"]
Sep 10 23:16:47.873977 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:47.873947    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-689fe3-29818035-tcg82"]
Sep 10 23:16:47.897126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:47.897095    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"]
Sep 10 23:16:47.897392 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:47.897341    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf" podUID="ae41f1ae-4afe-46d1-8081-71163b580cec" containerName="payload-processing" containerID="cri-o://1082444bccffad3f3d04b3404baaa993bededa29ceeb7f3b0a7f456e3196efc6" gracePeriod=30
Sep 10 23:16:47.904048 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:47.904016    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"]
Sep 10 23:16:47.904295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:47.904264    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt" podUID="98f639f0-5a65-4aea-a802-cb9fa8f76f4c" containerName="payload-pre-processing" containerID="cri-o://f1bfd978d873ab96bcb647867bd37d8f48a6d506321dbfdf4a04e5c399fcc4a8" gracePeriod=30
Sep 10 23:16:48.067581 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.067479    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7"]
Sep 10 23:16:48.073354 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.073252    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-689fe3-s7sh7"]
Sep 10 23:16:48.152147 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.152124    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"
Sep 10 23:16:48.196823 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.196798    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"
Sep 10 23:16:48.218689 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.218659    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f639f0-5a65-4aea-a802-cb9fa8f76f4c-plugins-config-volume\") pod \"98f639f0-5a65-4aea-a802-cb9fa8f76f4c\" (UID: \"98f639f0-5a65-4aea-a802-cb9fa8f76f4c\") "
Sep 10 23:16:48.218850 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.218724    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwjzw\" (UniqueName: \"kubernetes.io/projected/98f639f0-5a65-4aea-a802-cb9fa8f76f4c-kube-api-access-zwjzw\") pod \"98f639f0-5a65-4aea-a802-cb9fa8f76f4c\" (UID: \"98f639f0-5a65-4aea-a802-cb9fa8f76f4c\") "
Sep 10 23:16:48.218850 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.218753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ndrq\" (UniqueName: \"kubernetes.io/projected/ae41f1ae-4afe-46d1-8081-71163b580cec-kube-api-access-5ndrq\") pod \"ae41f1ae-4afe-46d1-8081-71163b580cec\" (UID: \"ae41f1ae-4afe-46d1-8081-71163b580cec\") "
Sep 10 23:16:48.218850 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.218814    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae41f1ae-4afe-46d1-8081-71163b580cec-plugins-config-volume\") pod \"ae41f1ae-4afe-46d1-8081-71163b580cec\" (UID: \"ae41f1ae-4afe-46d1-8081-71163b580cec\") "
Sep 10 23:16:48.219323 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.219069    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f639f0-5a65-4aea-a802-cb9fa8f76f4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98f639f0-5a65-4aea-a802-cb9fa8f76f4c" (UID: "98f639f0-5a65-4aea-a802-cb9fa8f76f4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:48.219323 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.219140    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f639f0-5a65-4aea-a802-cb9fa8f76f4c-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:48.219540 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.219476    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae41f1ae-4afe-46d1-8081-71163b580cec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae41f1ae-4afe-46d1-8081-71163b580cec" (UID: "ae41f1ae-4afe-46d1-8081-71163b580cec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:48.221416 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.221392    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae41f1ae-4afe-46d1-8081-71163b580cec-kube-api-access-5ndrq" (OuterVolumeSpecName: "kube-api-access-5ndrq") pod "ae41f1ae-4afe-46d1-8081-71163b580cec" (UID: "ae41f1ae-4afe-46d1-8081-71163b580cec"). InnerVolumeSpecName "kube-api-access-5ndrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:48.221527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.221470    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f639f0-5a65-4aea-a802-cb9fa8f76f4c-kube-api-access-zwjzw" (OuterVolumeSpecName: "kube-api-access-zwjzw") pod "98f639f0-5a65-4aea-a802-cb9fa8f76f4c" (UID: "98f639f0-5a65-4aea-a802-cb9fa8f76f4c"). InnerVolumeSpecName "kube-api-access-zwjzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:48.320071 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.320030    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwjzw\" (UniqueName: \"kubernetes.io/projected/98f639f0-5a65-4aea-a802-cb9fa8f76f4c-kube-api-access-zwjzw\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:48.320071 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.320064    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ndrq\" (UniqueName: \"kubernetes.io/projected/ae41f1ae-4afe-46d1-8081-71163b580cec-kube-api-access-5ndrq\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:48.320071 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.320078    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae41f1ae-4afe-46d1-8081-71163b580cec-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:48.367460 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.367423    2563 generic.go:356] "Generic (PLEG): container finished" podID="ae41f1ae-4afe-46d1-8081-71163b580cec" containerID="1082444bccffad3f3d04b3404baaa993bededa29ceeb7f3b0a7f456e3196efc6" exitCode=0
Sep 10 23:16:48.367638 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.367464    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"
Sep 10 23:16:48.367638 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.367473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf" event={"ID":"ae41f1ae-4afe-46d1-8081-71163b580cec","Type":"ContainerDied","Data":"1082444bccffad3f3d04b3404baaa993bededa29ceeb7f3b0a7f456e3196efc6"}
Sep 10 23:16:48.367638 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.367555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf" event={"ID":"ae41f1ae-4afe-46d1-8081-71163b580cec","Type":"ContainerDied","Data":"1cfc62e33752c6cd24a263228af952ce67cc992caed36dd7cc99b97a3167b8c9"}
Sep 10 23:16:48.367638 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.367582    2563 scope.go:117] "RemoveContainer" containerID="1082444bccffad3f3d04b3404baaa993bededa29ceeb7f3b0a7f456e3196efc6"
Sep 10 23:16:48.368837 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.368816    2563 generic.go:356] "Generic (PLEG): container finished" podID="98f639f0-5a65-4aea-a802-cb9fa8f76f4c" containerID="f1bfd978d873ab96bcb647867bd37d8f48a6d506321dbfdf4a04e5c399fcc4a8" exitCode=0
Sep 10 23:16:48.368923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.368857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt" event={"ID":"98f639f0-5a65-4aea-a802-cb9fa8f76f4c","Type":"ContainerDied","Data":"f1bfd978d873ab96bcb647867bd37d8f48a6d506321dbfdf4a04e5c399fcc4a8"}
Sep 10 23:16:48.368923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.368869    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"
Sep 10 23:16:48.368923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.368895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt" event={"ID":"98f639f0-5a65-4aea-a802-cb9fa8f76f4c","Type":"ContainerDied","Data":"b352ec6727b706a7f13c3eb7a3cf5cb016ebafa8125b4dd9f17308f5003e932f"}
Sep 10 23:16:48.381128 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.380974    2563 scope.go:117] "RemoveContainer" containerID="1082444bccffad3f3d04b3404baaa993bededa29ceeb7f3b0a7f456e3196efc6"
Sep 10 23:16:48.381475 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:16:48.381452    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1082444bccffad3f3d04b3404baaa993bededa29ceeb7f3b0a7f456e3196efc6\": container with ID starting with 1082444bccffad3f3d04b3404baaa993bededa29ceeb7f3b0a7f456e3196efc6 not found: ID does not exist" containerID="1082444bccffad3f3d04b3404baaa993bededa29ceeb7f3b0a7f456e3196efc6"
Sep 10 23:16:48.381570 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.381485    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1082444bccffad3f3d04b3404baaa993bededa29ceeb7f3b0a7f456e3196efc6"} err="failed to get container status \"1082444bccffad3f3d04b3404baaa993bededa29ceeb7f3b0a7f456e3196efc6\": rpc error: code = NotFound desc = could not find container \"1082444bccffad3f3d04b3404baaa993bededa29ceeb7f3b0a7f456e3196efc6\": container with ID starting with 1082444bccffad3f3d04b3404baaa993bededa29ceeb7f3b0a7f456e3196efc6 not found: ID does not exist"
Sep 10 23:16:48.381570 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.381533    2563 scope.go:117] "RemoveContainer" containerID="f1bfd978d873ab96bcb647867bd37d8f48a6d506321dbfdf4a04e5c399fcc4a8"
Sep 10 23:16:48.389817 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.389801    2563 scope.go:117] "RemoveContainer" containerID="f1bfd978d873ab96bcb647867bd37d8f48a6d506321dbfdf4a04e5c399fcc4a8"
Sep 10 23:16:48.390063 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:16:48.390046    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1bfd978d873ab96bcb647867bd37d8f48a6d506321dbfdf4a04e5c399fcc4a8\": container with ID starting with f1bfd978d873ab96bcb647867bd37d8f48a6d506321dbfdf4a04e5c399fcc4a8 not found: ID does not exist" containerID="f1bfd978d873ab96bcb647867bd37d8f48a6d506321dbfdf4a04e5c399fcc4a8"
Sep 10 23:16:48.390105 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.390069    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1bfd978d873ab96bcb647867bd37d8f48a6d506321dbfdf4a04e5c399fcc4a8"} err="failed to get container status \"f1bfd978d873ab96bcb647867bd37d8f48a6d506321dbfdf4a04e5c399fcc4a8\": rpc error: code = NotFound desc = could not find container \"f1bfd978d873ab96bcb647867bd37d8f48a6d506321dbfdf4a04e5c399fcc4a8\": container with ID starting with f1bfd978d873ab96bcb647867bd37d8f48a6d506321dbfdf4a04e5c399fcc4a8 not found: ID does not exist"
Sep 10 23:16:48.398045 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.398021    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"]
Sep 10 23:16:48.401761 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.401741    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-689fe3-fcf84fdf5-6nlkf"]
Sep 10 23:16:48.411588 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.411564    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"]
Sep 10 23:16:48.413414 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:48.413394    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-689fe3-6b7c8f9f46-q27xt"]
Sep 10 23:16:49.160473 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.160413    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"]
Sep 10 23:16:49.160946 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.160699    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z" podUID="dc0b3b13-5c79-47f1-a548-88da251fd1cf" containerName="payload-processing" containerID="cri-o://f92162cf806bc93d33037551f5ee5b87ba2bad7648aaa237926fcb1ef0cf5ff0" gracePeriod=30
Sep 10 23:16:49.336002 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.335960    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h"]
Sep 10 23:16:49.338093 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.338064    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-51383f0b-qrs5h"]
Sep 10 23:16:49.376746 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.376711    2563 generic.go:356] "Generic (PLEG): container finished" podID="dc0b3b13-5c79-47f1-a548-88da251fd1cf" containerID="f92162cf806bc93d33037551f5ee5b87ba2bad7648aaa237926fcb1ef0cf5ff0" exitCode=0
Sep 10 23:16:49.376870 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.376790    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z" event={"ID":"dc0b3b13-5c79-47f1-a548-88da251fd1cf","Type":"ContainerDied","Data":"f92162cf806bc93d33037551f5ee5b87ba2bad7648aaa237926fcb1ef0cf5ff0"}
Sep 10 23:16:49.408766 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.408740    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"
Sep 10 23:16:49.531519 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.531404    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwswd\" (UniqueName: \"kubernetes.io/projected/dc0b3b13-5c79-47f1-a548-88da251fd1cf-kube-api-access-bwswd\") pod \"dc0b3b13-5c79-47f1-a548-88da251fd1cf\" (UID: \"dc0b3b13-5c79-47f1-a548-88da251fd1cf\") "
Sep 10 23:16:49.531695 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.531543    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0b3b13-5c79-47f1-a548-88da251fd1cf-plugins-config-volume\") pod \"dc0b3b13-5c79-47f1-a548-88da251fd1cf\" (UID: \"dc0b3b13-5c79-47f1-a548-88da251fd1cf\") "
Sep 10 23:16:49.531925 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.531901    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc0b3b13-5c79-47f1-a548-88da251fd1cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dc0b3b13-5c79-47f1-a548-88da251fd1cf" (UID: "dc0b3b13-5c79-47f1-a548-88da251fd1cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:49.533657 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.533635    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0b3b13-5c79-47f1-a548-88da251fd1cf-kube-api-access-bwswd" (OuterVolumeSpecName: "kube-api-access-bwswd") pod "dc0b3b13-5c79-47f1-a548-88da251fd1cf" (UID: "dc0b3b13-5c79-47f1-a548-88da251fd1cf"). InnerVolumeSpecName "kube-api-access-bwswd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:49.632532 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.632452    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwswd\" (UniqueName: \"kubernetes.io/projected/dc0b3b13-5c79-47f1-a548-88da251fd1cf-kube-api-access-bwswd\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:49.632532 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.632527    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc0b3b13-5c79-47f1-a548-88da251fd1cf-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:49.814039 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.814002    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="144e80a1-38e0-4c86-85ee-222e0d249112" path="/var/lib/kubelet/pods/144e80a1-38e0-4c86-85ee-222e0d249112/volumes"
Sep 10 23:16:49.814334 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.814320    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25cc8b40-59aa-4695-a427-09d9b98a698e" path="/var/lib/kubelet/pods/25cc8b40-59aa-4695-a427-09d9b98a698e/volumes"
Sep 10 23:16:49.814662 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.814650    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98f639f0-5a65-4aea-a802-cb9fa8f76f4c" path="/var/lib/kubelet/pods/98f639f0-5a65-4aea-a802-cb9fa8f76f4c/volumes"
Sep 10 23:16:49.814964 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.814953    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae41f1ae-4afe-46d1-8081-71163b580cec" path="/var/lib/kubelet/pods/ae41f1ae-4afe-46d1-8081-71163b580cec/volumes"
Sep 10 23:16:49.815238 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:49.815227    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e89f7a2e-7339-4fa3-803c-be7d4ed348fd" path="/var/lib/kubelet/pods/e89f7a2e-7339-4fa3-803c-be7d4ed348fd/volumes"
Sep 10 23:16:50.384407 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:50.384374    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z" event={"ID":"dc0b3b13-5c79-47f1-a548-88da251fd1cf","Type":"ContainerDied","Data":"30f2997b97b4dec0fcb804e2080c4d11bbed841143206c71557793e439c24ab6"}
Sep 10 23:16:50.384407 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:50.384400    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"
Sep 10 23:16:50.384965 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:50.384432    2563 scope.go:117] "RemoveContainer" containerID="f92162cf806bc93d33037551f5ee5b87ba2bad7648aaa237926fcb1ef0cf5ff0"
Sep 10 23:16:50.404509 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:50.404472    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"]
Sep 10 23:16:50.411889 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:50.411867    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-51383f0b-589d4ffc85-bqg8z"]
Sep 10 23:16:51.814388 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:51.814356    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc0b3b13-5c79-47f1-a548-88da251fd1cf" path="/var/lib/kubelet/pods/dc0b3b13-5c79-47f1-a548-88da251fd1cf/volumes"
Sep 10 23:16:58.458156 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.458118    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv" podUID="45ffa2b2-1ad9-4384-8c83-e37076824c49" containerName="main" containerID="cri-o://cfc260f9bde2fa2d7d340eba5cd5d59d44a5486b22f1c7d1cadf147a5027a6b1" gracePeriod=45
Sep 10 23:16:58.711553 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.711472    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:58.813951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.813911    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-tmp-dir\") pod \"45ffa2b2-1ad9-4384-8c83-e37076824c49\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") "
Sep 10 23:16:58.814144 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.813960    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-model-cache\") pod \"45ffa2b2-1ad9-4384-8c83-e37076824c49\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") "
Sep 10 23:16:58.814144 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.813999    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-home\") pod \"45ffa2b2-1ad9-4384-8c83-e37076824c49\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") "
Sep 10 23:16:58.814144 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.814031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlncr\" (UniqueName: \"kubernetes.io/projected/45ffa2b2-1ad9-4384-8c83-e37076824c49-kube-api-access-hlncr\") pod \"45ffa2b2-1ad9-4384-8c83-e37076824c49\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") "
Sep 10 23:16:58.814144 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.814087    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-dshm\") pod \"45ffa2b2-1ad9-4384-8c83-e37076824c49\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") "
Sep 10 23:16:58.814144 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.814132    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45ffa2b2-1ad9-4384-8c83-e37076824c49-tls-certs\") pod \"45ffa2b2-1ad9-4384-8c83-e37076824c49\" (UID: \"45ffa2b2-1ad9-4384-8c83-e37076824c49\") "
Sep 10 23:16:58.814395 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.814187    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45ffa2b2-1ad9-4384-8c83-e37076824c49" (UID: "45ffa2b2-1ad9-4384-8c83-e37076824c49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:16:58.814395 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.814270    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-model-cache" (OuterVolumeSpecName: "model-cache") pod "45ffa2b2-1ad9-4384-8c83-e37076824c49" (UID: "45ffa2b2-1ad9-4384-8c83-e37076824c49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:16:58.814395 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.814281    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-home" (OuterVolumeSpecName: "home") pod "45ffa2b2-1ad9-4384-8c83-e37076824c49" (UID: "45ffa2b2-1ad9-4384-8c83-e37076824c49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:16:58.814562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.814476    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:58.814562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.814522    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:58.814562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.814532    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:58.816265 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.816230    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-dshm" (OuterVolumeSpecName: "dshm") pod "45ffa2b2-1ad9-4384-8c83-e37076824c49" (UID: "45ffa2b2-1ad9-4384-8c83-e37076824c49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:16:58.816386 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.816315    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ffa2b2-1ad9-4384-8c83-e37076824c49-kube-api-access-hlncr" (OuterVolumeSpecName: "kube-api-access-hlncr") pod "45ffa2b2-1ad9-4384-8c83-e37076824c49" (UID: "45ffa2b2-1ad9-4384-8c83-e37076824c49"). InnerVolumeSpecName "kube-api-access-hlncr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:58.816386 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.816328    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ffa2b2-1ad9-4384-8c83-e37076824c49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45ffa2b2-1ad9-4384-8c83-e37076824c49" (UID: "45ffa2b2-1ad9-4384-8c83-e37076824c49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:16:58.915353 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.915317    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlncr\" (UniqueName: \"kubernetes.io/projected/45ffa2b2-1ad9-4384-8c83-e37076824c49-kube-api-access-hlncr\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:58.915353 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.915347    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45ffa2b2-1ad9-4384-8c83-e37076824c49-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:58.915353 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:58.915358    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45ffa2b2-1ad9-4384-8c83-e37076824c49-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:16:59.434145 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:59.434114    2563 generic.go:356] "Generic (PLEG): container finished" podID="45ffa2b2-1ad9-4384-8c83-e37076824c49" containerID="cfc260f9bde2fa2d7d340eba5cd5d59d44a5486b22f1c7d1cadf147a5027a6b1" exitCode=0
Sep 10 23:16:59.434327 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:59.434153    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"
Sep 10 23:16:59.434327 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:59.434200    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv" event={"ID":"45ffa2b2-1ad9-4384-8c83-e37076824c49","Type":"ContainerDied","Data":"cfc260f9bde2fa2d7d340eba5cd5d59d44a5486b22f1c7d1cadf147a5027a6b1"}
Sep 10 23:16:59.434327 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:59.434241    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv" event={"ID":"45ffa2b2-1ad9-4384-8c83-e37076824c49","Type":"ContainerDied","Data":"fad09bf73d83e74757a613164d07595c38085251b62048deeaff7ac6e9b88ad1"}
Sep 10 23:16:59.434327 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:59.434258    2563 scope.go:117] "RemoveContainer" containerID="cfc260f9bde2fa2d7d340eba5cd5d59d44a5486b22f1c7d1cadf147a5027a6b1"
Sep 10 23:16:59.443613 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:59.443590    2563 scope.go:117] "RemoveContainer" containerID="cfc260f9bde2fa2d7d340eba5cd5d59d44a5486b22f1c7d1cadf147a5027a6b1"
Sep 10 23:16:59.443906 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:16:59.443885    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc260f9bde2fa2d7d340eba5cd5d59d44a5486b22f1c7d1cadf147a5027a6b1\": container with ID starting with cfc260f9bde2fa2d7d340eba5cd5d59d44a5486b22f1c7d1cadf147a5027a6b1 not found: ID does not exist" containerID="cfc260f9bde2fa2d7d340eba5cd5d59d44a5486b22f1c7d1cadf147a5027a6b1"
Sep 10 23:16:59.443954 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:59.443915    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc260f9bde2fa2d7d340eba5cd5d59d44a5486b22f1c7d1cadf147a5027a6b1"} err="failed to get container status \"cfc260f9bde2fa2d7d340eba5cd5d59d44a5486b22f1c7d1cadf147a5027a6b1\": rpc error: code = NotFound desc = could not find container \"cfc260f9bde2fa2d7d340eba5cd5d59d44a5486b22f1c7d1cadf147a5027a6b1\": container with ID starting with cfc260f9bde2fa2d7d340eba5cd5d59d44a5486b22f1c7d1cadf147a5027a6b1 not found: ID does not exist"
Sep 10 23:16:59.454756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:59.454732    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"]
Sep 10 23:16:59.458061 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:59.458038    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-689fe3/autores-model-689fe3-kserve-d5b47cb6c-p7mbv"]
Sep 10 23:16:59.813969 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:16:59.813942    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ffa2b2-1ad9-4384-8c83-e37076824c49" path="/var/lib/kubelet/pods/45ffa2b2-1ad9-4384-8c83-e37076824c49/volumes"
Sep 10 23:17:24.450729 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.450693    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"]
Sep 10 23:17:24.451310 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451289    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc0b3b13-5c79-47f1-a548-88da251fd1cf" containerName="payload-processing"
Sep 10 23:17:24.451367 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451314    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc0b3b13-5c79-47f1-a548-88da251fd1cf" containerName="payload-processing"
Sep 10 23:17:24.451367 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451327    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45ffa2b2-1ad9-4384-8c83-e37076824c49" containerName="main"
Sep 10 23:17:24.451367 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451335    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="45ffa2b2-1ad9-4384-8c83-e37076824c49" containerName="main"
Sep 10 23:17:24.451367 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451361    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25cc8b40-59aa-4695-a427-09d9b98a698e" containerName="revoke-keys"
Sep 10 23:17:24.451511 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451369    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25cc8b40-59aa-4695-a427-09d9b98a698e" containerName="revoke-keys"
Sep 10 23:17:24.451511 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451382    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae41f1ae-4afe-46d1-8081-71163b580cec" containerName="payload-processing"
Sep 10 23:17:24.451511 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451389    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae41f1ae-4afe-46d1-8081-71163b580cec" containerName="payload-processing"
Sep 10 23:17:24.451511 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451404    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f639f0-5a65-4aea-a802-cb9fa8f76f4c" containerName="payload-pre-processing"
Sep 10 23:17:24.451511 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451412    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f639f0-5a65-4aea-a802-cb9fa8f76f4c" containerName="payload-pre-processing"
Sep 10 23:17:24.451671 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451518    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f639f0-5a65-4aea-a802-cb9fa8f76f4c" containerName="payload-pre-processing"
Sep 10 23:17:24.451671 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451532    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="45ffa2b2-1ad9-4384-8c83-e37076824c49" containerName="main"
Sep 10 23:17:24.451671 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451542    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae41f1ae-4afe-46d1-8081-71163b580cec" containerName="payload-processing"
Sep 10 23:17:24.451671 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451551    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25cc8b40-59aa-4695-a427-09d9b98a698e" containerName="revoke-keys"
Sep 10 23:17:24.451671 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.451561    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc0b3b13-5c79-47f1-a548-88da251fd1cf" containerName="payload-processing"
Sep 10 23:17:24.456018 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.455999    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.458405 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.458384    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"premium-w4-7025b9-kserve-self-signed-certs\""
Sep 10 23:17:24.465478 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.465450    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"]
Sep 10 23:17:24.553589 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.553557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x8lb\" (UniqueName: \"kubernetes.io/projected/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-kube-api-access-4x8lb\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.553742 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.553603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-dshm\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.553742 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.553629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-model-cache\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.553742 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.553683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-home\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.553742 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.553732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-tls-certs\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.553887 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.553755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-tmp-dir\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.654922 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.654885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-tls-certs\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.654922 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.654925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-tmp-dir\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.655178 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.654970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x8lb\" (UniqueName: \"kubernetes.io/projected/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-kube-api-access-4x8lb\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.655178 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.655005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-dshm\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.655178 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.655030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-model-cache\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.655178 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.655068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-home\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.655410 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.655336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-tmp-dir\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.655410 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.655391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-home\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.655531 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.655481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-model-cache\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.657366 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.657344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-dshm\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.657466 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.657449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-tls-certs\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.662955 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.662931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x8lb\" (UniqueName: \"kubernetes.io/projected/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-kube-api-access-4x8lb\") pod \"premium-w4-7025b9-kserve-7f98995769-sbbf4\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") " pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:24.765819 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:24.765723    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:25.038171 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:25.038144    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"]
Sep 10 23:17:25.039890 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:17:25.039859    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde1d1d9c_4d22_454d_8e6a_4d3c696af3d9.slice/crio-84cb7b028823dad9480d28fc8652f535d9f689f9a349c64f118f0a3ef7a8d12c WatchSource:0}: Error finding container 84cb7b028823dad9480d28fc8652f535d9f689f9a349c64f118f0a3ef7a8d12c: Status 404 returned error can't find the container with id 84cb7b028823dad9480d28fc8652f535d9f689f9a349c64f118f0a3ef7a8d12c
Sep 10 23:17:25.576634 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:25.576592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4" event={"ID":"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9","Type":"ContainerStarted","Data":"bf1b4de9853c920d9a3652aa774340b043a0fa473310b5943cca6092fbce5ad4"}
Sep 10 23:17:25.577034 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:25.576647    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4" event={"ID":"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9","Type":"ContainerStarted","Data":"84cb7b028823dad9480d28fc8652f535d9f689f9a349c64f118f0a3ef7a8d12c"}
Sep 10 23:17:25.595538 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:25.595469    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4" podStartSLOduration=1.595452593 podStartE2EDuration="1.595452593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:17:25.593673607 +0000 UTC m=+966.285250802" watchObservedRunningTime="2026-09-10 23:17:25.595452593 +0000 UTC m=+966.287029777"
Sep 10 23:17:34.766153 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:34.766097    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:34.766682 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:34.766179    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:34.778814 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:34.778789    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:35.639891 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:35.639862    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:17:37.278836 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.278803    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"]
Sep 10 23:17:37.282775 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.282759    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"
Sep 10 23:17:37.284958 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.284931    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b3d9be\""
Sep 10 23:17:37.284958 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.284945    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b3d9be-dockercfg-qg27x\""
Sep 10 23:17:37.293018 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.292993    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"]
Sep 10 23:17:37.309927 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.309895    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"]
Sep 10 23:17:37.313817 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.313798    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"
Sep 10 23:17:37.322661 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.322639    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"]
Sep 10 23:17:37.374910 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.374878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c7d380-0d07-4bd8-a461-357fa7e09964-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n\" (UID: \"c3c7d380-0d07-4bd8-a461-357fa7e09964\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"
Sep 10 23:17:37.375098 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.374930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-676w4\" (UniqueName: \"kubernetes.io/projected/c3c7d380-0d07-4bd8-a461-357fa7e09964-kube-api-access-676w4\") pod \"payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n\" (UID: \"c3c7d380-0d07-4bd8-a461-357fa7e09964\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"
Sep 10 23:17:37.375098 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.374952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g47wh\" (UniqueName: \"kubernetes.io/projected/862ad768-cdf7-4b2d-8601-02d43f4fdab6-kube-api-access-g47wh\") pod \"payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx\" (UID: \"862ad768-cdf7-4b2d-8601-02d43f4fdab6\") " pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"
Sep 10 23:17:37.375098 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.375045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862ad768-cdf7-4b2d-8601-02d43f4fdab6-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx\" (UID: \"862ad768-cdf7-4b2d-8601-02d43f4fdab6\") " pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"
Sep 10 23:17:37.475709 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.475675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c7d380-0d07-4bd8-a461-357fa7e09964-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n\" (UID: \"c3c7d380-0d07-4bd8-a461-357fa7e09964\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"
Sep 10 23:17:37.475892 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.475729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676w4\" (UniqueName: \"kubernetes.io/projected/c3c7d380-0d07-4bd8-a461-357fa7e09964-kube-api-access-676w4\") pod \"payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n\" (UID: \"c3c7d380-0d07-4bd8-a461-357fa7e09964\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"
Sep 10 23:17:37.475892 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.475759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g47wh\" (UniqueName: \"kubernetes.io/projected/862ad768-cdf7-4b2d-8601-02d43f4fdab6-kube-api-access-g47wh\") pod \"payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx\" (UID: \"862ad768-cdf7-4b2d-8601-02d43f4fdab6\") " pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"
Sep 10 23:17:37.475892 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.475791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862ad768-cdf7-4b2d-8601-02d43f4fdab6-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx\" (UID: \"862ad768-cdf7-4b2d-8601-02d43f4fdab6\") " pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"
Sep 10 23:17:37.476385 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.476364    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c7d380-0d07-4bd8-a461-357fa7e09964-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n\" (UID: \"c3c7d380-0d07-4bd8-a461-357fa7e09964\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"
Sep 10 23:17:37.476452 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.476418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862ad768-cdf7-4b2d-8601-02d43f4fdab6-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx\" (UID: \"862ad768-cdf7-4b2d-8601-02d43f4fdab6\") " pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"
Sep 10 23:17:37.486713 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.486688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g47wh\" (UniqueName: \"kubernetes.io/projected/862ad768-cdf7-4b2d-8601-02d43f4fdab6-kube-api-access-g47wh\") pod \"payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx\" (UID: \"862ad768-cdf7-4b2d-8601-02d43f4fdab6\") " pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"
Sep 10 23:17:37.490286 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.490266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-676w4\" (UniqueName: \"kubernetes.io/projected/c3c7d380-0d07-4bd8-a461-357fa7e09964-kube-api-access-676w4\") pod \"payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n\" (UID: \"c3c7d380-0d07-4bd8-a461-357fa7e09964\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"
Sep 10 23:17:37.594486 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.594453    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"
Sep 10 23:17:37.624590 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.624555    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"
Sep 10 23:17:37.872153 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.872111    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"]
Sep 10 23:17:37.874383 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:17:37.874360    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3c7d380_0d07_4bd8_a461_357fa7e09964.slice/crio-1f5bb6407a550addfa3b3704e0e43daa7538b469daac8880814e6a1e9a6197fc WatchSource:0}: Error finding container 1f5bb6407a550addfa3b3704e0e43daa7538b469daac8880814e6a1e9a6197fc: Status 404 returned error can't find the container with id 1f5bb6407a550addfa3b3704e0e43daa7538b469daac8880814e6a1e9a6197fc
Sep 10 23:17:37.901981 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:37.901959    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"]
Sep 10 23:17:37.907629 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:17:37.907595    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862ad768_cdf7_4b2d_8601_02d43f4fdab6.slice/crio-c0154066e1ef8a22fccc4c3f277cb501be62675af57d2f35adabc01d47a5bfdb WatchSource:0}: Error finding container c0154066e1ef8a22fccc4c3f277cb501be62675af57d2f35adabc01d47a5bfdb: Status 404 returned error can't find the container with id c0154066e1ef8a22fccc4c3f277cb501be62675af57d2f35adabc01d47a5bfdb
Sep 10 23:17:38.647062 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:38.647031    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n" event={"ID":"c3c7d380-0d07-4bd8-a461-357fa7e09964","Type":"ContainerStarted","Data":"cec45fbd0ce6335de7ee93f5c50ebf511a5b049072b9a850dfe69bca08a4c907"}
Sep 10 23:17:38.647524 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:38.647081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n" event={"ID":"c3c7d380-0d07-4bd8-a461-357fa7e09964","Type":"ContainerStarted","Data":"1f5bb6407a550addfa3b3704e0e43daa7538b469daac8880814e6a1e9a6197fc"}
Sep 10 23:17:38.647524 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:38.647207    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"
Sep 10 23:17:38.648684 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:38.648663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx" event={"ID":"862ad768-cdf7-4b2d-8601-02d43f4fdab6","Type":"ContainerStarted","Data":"37e5be107009904f3f53c90a908ad120a6a87ffee2abcd4262af73607e2f8ba0"}
Sep 10 23:17:38.648786 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:38.648697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx" event={"ID":"862ad768-cdf7-4b2d-8601-02d43f4fdab6","Type":"ContainerStarted","Data":"c0154066e1ef8a22fccc4c3f277cb501be62675af57d2f35adabc01d47a5bfdb"}
Sep 10 23:17:38.648786 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:38.648778    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"
Sep 10 23:17:38.677228 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:38.677183    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n" podStartSLOduration=1.677169812 podStartE2EDuration="1.677169812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:17:38.675138083 +0000 UTC m=+979.366715294" watchObservedRunningTime="2026-09-10 23:17:38.677169812 +0000 UTC m=+979.368747001"
Sep 10 23:17:38.693907 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:38.693870    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx" podStartSLOduration=1.6938566659999998 podStartE2EDuration="1.693856666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:17:38.692685811 +0000 UTC m=+979.384263017" watchObservedRunningTime="2026-09-10 23:17:38.693856666 +0000 UTC m=+979.385433857"
Sep 10 23:17:44.936951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:44.936876    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"]
Sep 10 23:17:44.940762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:44.940742    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:44.943217 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:44.943194    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ce17e\"/\"kube-root-ca.crt\""
Sep 10 23:17:44.943328 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:44.943195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce17e\"/\"default-dockercfg-h8dqd\""
Sep 10 23:17:44.944047 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:44.944021    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ce17e\"/\"openshift-service-ca.crt\""
Sep 10 23:17:44.944180 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:44.944157    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce17e\"/\"premium-w0-6ce17e-kserve-self-signed-certs\""
Sep 10 23:17:44.952059 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:44.952038    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"]
Sep 10 23:17:45.045705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.045673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-home\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.045874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.045722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e515df-f4c6-4952-abd4-5421450cfdfd-tls-certs\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.045874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.045748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-dshm\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.045874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.045769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfd7d\" (UniqueName: \"kubernetes.io/projected/32e515df-f4c6-4952-abd4-5421450cfdfd-kube-api-access-tfd7d\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.045985 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.045908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-model-cache\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.045985 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.045940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-tmp-dir\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.146951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.146918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-dshm\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.146951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.146957    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfd7d\" (UniqueName: \"kubernetes.io/projected/32e515df-f4c6-4952-abd4-5421450cfdfd-kube-api-access-tfd7d\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.147152 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.147041    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-model-cache\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.147191 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.147163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-tmp-dir\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.147227 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.147216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-home\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.147262 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.147246    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e515df-f4c6-4952-abd4-5421450cfdfd-tls-certs\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.147419 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.147400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-model-cache\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.147598 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.147549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-tmp-dir\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.147598 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.147558    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-home\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.149217 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.149191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-dshm\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.149573 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.149556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e515df-f4c6-4952-abd4-5421450cfdfd-tls-certs\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.155234 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.155214    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfd7d\" (UniqueName: \"kubernetes.io/projected/32e515df-f4c6-4952-abd4-5421450cfdfd-kube-api-access-tfd7d\") pod \"premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") " pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.251676 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.251589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:45.527516 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.527401    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"]
Sep 10 23:17:45.532947 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:17:45.532920    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32e515df_f4c6_4952_abd4_5421450cfdfd.slice/crio-526740c01e2df35231139fa1871dc28fd1da509a14cfbf7962c4fa24187ffd64 WatchSource:0}: Error finding container 526740c01e2df35231139fa1871dc28fd1da509a14cfbf7962c4fa24187ffd64: Status 404 returned error can't find the container with id 526740c01e2df35231139fa1871dc28fd1da509a14cfbf7962c4fa24187ffd64
Sep 10 23:17:45.686186 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.686146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92" event={"ID":"32e515df-f4c6-4952-abd4-5421450cfdfd","Type":"ContainerStarted","Data":"b21d05bd3357c6f95e2a30a11435f39d156df8e83043516c788b8f52e5b56db6"}
Sep 10 23:17:45.686376 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.686194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92" event={"ID":"32e515df-f4c6-4952-abd4-5421450cfdfd","Type":"ContainerStarted","Data":"526740c01e2df35231139fa1871dc28fd1da509a14cfbf7962c4fa24187ffd64"}
Sep 10 23:17:45.702478 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:45.702249    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92" podStartSLOduration=1.702228668 podStartE2EDuration="1.702228668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:17:45.701572244 +0000 UTC m=+986.393149447" watchObservedRunningTime="2026-09-10 23:17:45.702228668 +0000 UTC m=+986.393805856"
Sep 10 23:17:49.655556 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:49.655519    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"
Sep 10 23:17:49.655957 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:49.655736    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"
Sep 10 23:17:55.252003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:55.251965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:55.252003 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:55.252011    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:55.264517 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:55.264472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:17:55.752424 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:17:55.752389    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:19:03.819409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:03.819374    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6"]
Sep 10 23:19:03.823039 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:03.823022    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6"
Sep 10 23:19:03.825530 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:03.825507    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:19:03.825638 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:03.825485    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:19:03.830239 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:03.830207    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6"]
Sep 10 23:19:03.903286 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:03.903252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eff5d87-467b-4b50-8b5a-cd1d27223815-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6\" (UID: \"4eff5d87-467b-4b50-8b5a-cd1d27223815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6"
Sep 10 23:19:04.004663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:04.004608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eff5d87-467b-4b50-8b5a-cd1d27223815-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6\" (UID: \"4eff5d87-467b-4b50-8b5a-cd1d27223815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6"
Sep 10 23:19:04.005255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:04.005232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eff5d87-467b-4b50-8b5a-cd1d27223815-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6\" (UID: \"4eff5d87-467b-4b50-8b5a-cd1d27223815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6"
Sep 10 23:19:04.135539 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:04.135432    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6"
Sep 10 23:19:04.405186 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:04.405156    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6"]
Sep 10 23:19:04.407860 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:19:04.407828    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eff5d87_467b_4b50_8b5a_cd1d27223815.slice/crio-10f55b59b372f9619f97f68e15e782e81572e0a835cb917f549cb5b80dbf320d WatchSource:0}: Error finding container 10f55b59b372f9619f97f68e15e782e81572e0a835cb917f549cb5b80dbf320d: Status 404 returned error can't find the container with id 10f55b59b372f9619f97f68e15e782e81572e0a835cb917f549cb5b80dbf320d
Sep 10 23:19:05.130000 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.129962    2563 generic.go:356] "Generic (PLEG): container finished" podID="4eff5d87-467b-4b50-8b5a-cd1d27223815" containerID="9c1bb5978b6a5674f157ee27b38549b381dca02673ab4a688207232a5c50e4c3" exitCode=0
Sep 10 23:19:05.130380 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.130045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6" event={"ID":"4eff5d87-467b-4b50-8b5a-cd1d27223815","Type":"ContainerDied","Data":"9c1bb5978b6a5674f157ee27b38549b381dca02673ab4a688207232a5c50e4c3"}
Sep 10 23:19:05.130380 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.130097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6" event={"ID":"4eff5d87-467b-4b50-8b5a-cd1d27223815","Type":"ContainerStarted","Data":"10f55b59b372f9619f97f68e15e782e81572e0a835cb917f549cb5b80dbf320d"}
Sep 10 23:19:05.248664 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.248630    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"]
Sep 10 23:19:05.252356 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.252334    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.254794 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.254772    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b3d9be\"/\"openshift-service-ca.crt\""
Sep 10 23:19:05.254920 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.254813    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b3d9be\"/\"explicit-model-b3d9be-kserve-self-signed-certs\""
Sep 10 23:19:05.254920 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.254835    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b3d9be\"/\"kube-root-ca.crt\""
Sep 10 23:19:05.255049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.254930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b3d9be\"/\"default-dockercfg-2xzmq\""
Sep 10 23:19:05.260847 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.260824    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"]
Sep 10 23:19:05.417451 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.417421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-tmp-dir\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.417663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.417467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20df8786-fdda-490f-8971-5dbb554d9f29-tls-certs\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.417663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.417517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcjv2\" (UniqueName: \"kubernetes.io/projected/20df8786-fdda-490f-8971-5dbb554d9f29-kube-api-access-tcjv2\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.417663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.417538    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-home\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.417663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.417581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-model-cache\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.417865 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.417679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-dshm\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.518974 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.518941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-dshm\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.519167 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.519000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-tmp-dir\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.519167 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.519031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20df8786-fdda-490f-8971-5dbb554d9f29-tls-certs\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.519167 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.519057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcjv2\" (UniqueName: \"kubernetes.io/projected/20df8786-fdda-490f-8971-5dbb554d9f29-kube-api-access-tcjv2\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.519167 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.519074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-home\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.519167 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.519119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-model-cache\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.519535 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.519511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-tmp-dir\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.519618 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.519535    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-model-cache\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.519618 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.519581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-home\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.521344 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.521325    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-dshm\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.521547 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.521533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20df8786-fdda-490f-8971-5dbb554d9f29-tls-certs\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.526648 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.526628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcjv2\" (UniqueName: \"kubernetes.io/projected/20df8786-fdda-490f-8971-5dbb554d9f29-kube-api-access-tcjv2\") pod \"explicit-model-b3d9be-kserve-86999d748c-6p54n\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") " pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.563180 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.563154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:05.835434 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:05.835411    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"]
Sep 10 23:19:05.837304 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:19:05.837277    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20df8786_fdda_490f_8971_5dbb554d9f29.slice/crio-6f42e4c1c9aea513d46cc252f11bd5a94c40399796a152e91a90e31908ee1254 WatchSource:0}: Error finding container 6f42e4c1c9aea513d46cc252f11bd5a94c40399796a152e91a90e31908ee1254: Status 404 returned error can't find the container with id 6f42e4c1c9aea513d46cc252f11bd5a94c40399796a152e91a90e31908ee1254
Sep 10 23:19:06.136957 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:06.136865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n" event={"ID":"20df8786-fdda-490f-8971-5dbb554d9f29","Type":"ContainerStarted","Data":"ccda55bd135a6d10d4b7be85faad76a8151f3ccbc1e72c43d291e34b6786a5b6"}
Sep 10 23:19:06.136957 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:06.136913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n" event={"ID":"20df8786-fdda-490f-8971-5dbb554d9f29","Type":"ContainerStarted","Data":"6f42e4c1c9aea513d46cc252f11bd5a94c40399796a152e91a90e31908ee1254"}
Sep 10 23:19:06.153929 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:06.153880    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n" podStartSLOduration=1.153863641 podStartE2EDuration="1.153863641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:19:06.151567022 +0000 UTC m=+1066.843144230" watchObservedRunningTime="2026-09-10 23:19:06.153863641 +0000 UTC m=+1066.845440825"
Sep 10 23:19:06.263286 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:06.263261    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6"
Sep 10 23:19:06.431688 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:06.431596    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eff5d87-467b-4b50-8b5a-cd1d27223815-maas-api-service-ca\") pod \"4eff5d87-467b-4b50-8b5a-cd1d27223815\" (UID: \"4eff5d87-467b-4b50-8b5a-cd1d27223815\") "
Sep 10 23:19:06.431941 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:06.431918    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eff5d87-467b-4b50-8b5a-cd1d27223815-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4eff5d87-467b-4b50-8b5a-cd1d27223815" (UID: "4eff5d87-467b-4b50-8b5a-cd1d27223815"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:19:06.532640 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:06.532612    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eff5d87-467b-4b50-8b5a-cd1d27223815-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:19:07.150595 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:07.150560    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6"
Sep 10 23:19:07.150595 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:07.150588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6" event={"ID":"4eff5d87-467b-4b50-8b5a-cd1d27223815","Type":"ContainerDied","Data":"10f55b59b372f9619f97f68e15e782e81572e0a835cb917f549cb5b80dbf320d"}
Sep 10 23:19:07.151048 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:07.150656    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10f55b59b372f9619f97f68e15e782e81572e0a835cb917f549cb5b80dbf320d"
Sep 10 23:19:14.541137 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:14.541036    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6"]
Sep 10 23:19:14.543263 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:14.543240    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d061fe3-6k2f6"]
Sep 10 23:19:15.563728 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:15.563679    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:15.564177 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:15.563746    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:15.576746 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:15.576715    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:15.814204 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:15.814125    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eff5d87-467b-4b50-8b5a-cd1d27223815" path="/var/lib/kubelet/pods/4eff5d87-467b-4b50-8b5a-cd1d27223815/volumes"
Sep 10 23:19:16.211603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:16.211486    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:19:19.135276 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.135246    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"]
Sep 10 23:19:19.135747 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.135733    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eff5d87-467b-4b50-8b5a-cd1d27223815" containerName="revoke-keys"
Sep 10 23:19:19.135747 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.135749    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eff5d87-467b-4b50-8b5a-cd1d27223815" containerName="revoke-keys"
Sep 10 23:19:19.135839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.135830    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eff5d87-467b-4b50-8b5a-cd1d27223815" containerName="revoke-keys"
Sep 10 23:19:19.138880 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.138863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.141339 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.141315    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"distinct-w4-7025b9-kserve-self-signed-certs\""
Sep 10 23:19:19.149646 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.149620    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"]
Sep 10 23:19:19.159570 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.159549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-dshm\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.159687 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.159620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdstr\" (UniqueName: \"kubernetes.io/projected/12269fd5-19ad-4396-9387-7e3c56ef50e7-kube-api-access-wdstr\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.159687 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.159667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-model-cache\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.159860 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.159730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12269fd5-19ad-4396-9387-7e3c56ef50e7-tls-certs\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.159860 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.159807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-home\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.159860 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.159835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-tmp-dir\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.260686 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.260650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-dshm\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.260901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.260733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdstr\" (UniqueName: \"kubernetes.io/projected/12269fd5-19ad-4396-9387-7e3c56ef50e7-kube-api-access-wdstr\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.260901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.260767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-model-cache\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.260901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.260796    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12269fd5-19ad-4396-9387-7e3c56ef50e7-tls-certs\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.260901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.260872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-home\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.260901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.260896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-tmp-dir\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.261176 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.261148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-model-cache\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.261239 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.261180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-home\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.261293 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.261246    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-tmp-dir\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.262960 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.262940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-dshm\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.263366 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.263345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12269fd5-19ad-4396-9387-7e3c56ef50e7-tls-certs\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.268516 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.268483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdstr\" (UniqueName: \"kubernetes.io/projected/12269fd5-19ad-4396-9387-7e3c56ef50e7-kube-api-access-wdstr\") pod \"distinct-w4-7025b9-kserve-68585676f6-k2w6p\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.448765 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.448673    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:19.719334 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:19.719305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"]
Sep 10 23:19:19.721572 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:19:19.721547    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12269fd5_19ad_4396_9387_7e3c56ef50e7.slice/crio-8e5e1d3b6e32e99d24c96669ccf4dfface6c3580cf91d374910b4edf250f2df4 WatchSource:0}: Error finding container 8e5e1d3b6e32e99d24c96669ccf4dfface6c3580cf91d374910b4edf250f2df4: Status 404 returned error can't find the container with id 8e5e1d3b6e32e99d24c96669ccf4dfface6c3580cf91d374910b4edf250f2df4
Sep 10 23:19:20.226973 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:20.226938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p" event={"ID":"12269fd5-19ad-4396-9387-7e3c56ef50e7","Type":"ContainerStarted","Data":"f9bff82d172da7b8999f6de38e295e2595de4ffe4833b1e58d73383ad6bc2750"}
Sep 10 23:19:20.226973 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:20.226981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p" event={"ID":"12269fd5-19ad-4396-9387-7e3c56ef50e7","Type":"ContainerStarted","Data":"8e5e1d3b6e32e99d24c96669ccf4dfface6c3580cf91d374910b4edf250f2df4"}
Sep 10 23:19:20.244440 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:20.244394    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p" podStartSLOduration=1.24437932 podStartE2EDuration="1.24437932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:19:20.244045175 +0000 UTC m=+1080.935622395" watchObservedRunningTime="2026-09-10 23:19:20.24437932 +0000 UTC m=+1080.935956504"
Sep 10 23:19:29.449690 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:29.449652    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:29.449690 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:29.449696    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:29.462476 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:29.462453    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:30.299777 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:30.299746    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:19:48.581620 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:48.581580    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"]
Sep 10 23:19:48.582068 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:48.581852    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx" podUID="862ad768-cdf7-4b2d-8601-02d43f4fdab6" containerName="payload-processing" containerID="cri-o://37e5be107009904f3f53c90a908ad120a6a87ffee2abcd4262af73607e2f8ba0" gracePeriod=30
Sep 10 23:19:48.587754 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:48.587718    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"]
Sep 10 23:19:48.588067 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:48.587966    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n" podUID="c3c7d380-0d07-4bd8-a461-357fa7e09964" containerName="payload-pre-processing" containerID="cri-o://cec45fbd0ce6335de7ee93f5c50ebf511a5b049072b9a850dfe69bca08a4c907" gracePeriod=30
Sep 10 23:19:48.859019 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:48.858995    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"
Sep 10 23:19:48.885765 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:48.885739    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"
Sep 10 23:19:48.946357 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:48.946328    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g47wh\" (UniqueName: \"kubernetes.io/projected/862ad768-cdf7-4b2d-8601-02d43f4fdab6-kube-api-access-g47wh\") pod \"862ad768-cdf7-4b2d-8601-02d43f4fdab6\" (UID: \"862ad768-cdf7-4b2d-8601-02d43f4fdab6\") "
Sep 10 23:19:48.946562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:48.946380    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862ad768-cdf7-4b2d-8601-02d43f4fdab6-plugins-config-volume\") pod \"862ad768-cdf7-4b2d-8601-02d43f4fdab6\" (UID: \"862ad768-cdf7-4b2d-8601-02d43f4fdab6\") "
Sep 10 23:19:48.946808 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:48.946779    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862ad768-cdf7-4b2d-8601-02d43f4fdab6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "862ad768-cdf7-4b2d-8601-02d43f4fdab6" (UID: "862ad768-cdf7-4b2d-8601-02d43f4fdab6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:19:48.948377 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:48.948359    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862ad768-cdf7-4b2d-8601-02d43f4fdab6-kube-api-access-g47wh" (OuterVolumeSpecName: "kube-api-access-g47wh") pod "862ad768-cdf7-4b2d-8601-02d43f4fdab6" (UID: "862ad768-cdf7-4b2d-8601-02d43f4fdab6"). InnerVolumeSpecName "kube-api-access-g47wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:19:49.047403 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.047371    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-676w4\" (UniqueName: \"kubernetes.io/projected/c3c7d380-0d07-4bd8-a461-357fa7e09964-kube-api-access-676w4\") pod \"c3c7d380-0d07-4bd8-a461-357fa7e09964\" (UID: \"c3c7d380-0d07-4bd8-a461-357fa7e09964\") "
Sep 10 23:19:49.047616 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.047459    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c7d380-0d07-4bd8-a461-357fa7e09964-plugins-config-volume\") pod \"c3c7d380-0d07-4bd8-a461-357fa7e09964\" (UID: \"c3c7d380-0d07-4bd8-a461-357fa7e09964\") "
Sep 10 23:19:49.047964 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.047848    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g47wh\" (UniqueName: \"kubernetes.io/projected/862ad768-cdf7-4b2d-8601-02d43f4fdab6-kube-api-access-g47wh\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:19:49.047964 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.047873    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862ad768-cdf7-4b2d-8601-02d43f4fdab6-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:19:49.048118 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.047971    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3c7d380-0d07-4bd8-a461-357fa7e09964-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3c7d380-0d07-4bd8-a461-357fa7e09964" (UID: "c3c7d380-0d07-4bd8-a461-357fa7e09964"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:19:49.049558 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.049531    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c7d380-0d07-4bd8-a461-357fa7e09964-kube-api-access-676w4" (OuterVolumeSpecName: "kube-api-access-676w4") pod "c3c7d380-0d07-4bd8-a461-357fa7e09964" (UID: "c3c7d380-0d07-4bd8-a461-357fa7e09964"). InnerVolumeSpecName "kube-api-access-676w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:19:49.148387 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.148314    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3c7d380-0d07-4bd8-a461-357fa7e09964-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:19:49.148387 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.148339    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-676w4\" (UniqueName: \"kubernetes.io/projected/c3c7d380-0d07-4bd8-a461-357fa7e09964-kube-api-access-676w4\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:19:49.396092 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.396053    2563 generic.go:356] "Generic (PLEG): container finished" podID="862ad768-cdf7-4b2d-8601-02d43f4fdab6" containerID="37e5be107009904f3f53c90a908ad120a6a87ffee2abcd4262af73607e2f8ba0" exitCode=0
Sep 10 23:19:49.396266 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.396098    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"
Sep 10 23:19:49.396266 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.396136    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx" event={"ID":"862ad768-cdf7-4b2d-8601-02d43f4fdab6","Type":"ContainerDied","Data":"37e5be107009904f3f53c90a908ad120a6a87ffee2abcd4262af73607e2f8ba0"}
Sep 10 23:19:49.396266 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.396190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx" event={"ID":"862ad768-cdf7-4b2d-8601-02d43f4fdab6","Type":"ContainerDied","Data":"c0154066e1ef8a22fccc4c3f277cb501be62675af57d2f35adabc01d47a5bfdb"}
Sep 10 23:19:49.396266 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.396250    2563 scope.go:117] "RemoveContainer" containerID="37e5be107009904f3f53c90a908ad120a6a87ffee2abcd4262af73607e2f8ba0"
Sep 10 23:19:49.397530 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.397481    2563 generic.go:356] "Generic (PLEG): container finished" podID="c3c7d380-0d07-4bd8-a461-357fa7e09964" containerID="cec45fbd0ce6335de7ee93f5c50ebf511a5b049072b9a850dfe69bca08a4c907" exitCode=0
Sep 10 23:19:49.397530 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.397520    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n" event={"ID":"c3c7d380-0d07-4bd8-a461-357fa7e09964","Type":"ContainerDied","Data":"cec45fbd0ce6335de7ee93f5c50ebf511a5b049072b9a850dfe69bca08a4c907"}
Sep 10 23:19:49.397709 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.397543    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"
Sep 10 23:19:49.397709 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.397570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n" event={"ID":"c3c7d380-0d07-4bd8-a461-357fa7e09964","Type":"ContainerDied","Data":"1f5bb6407a550addfa3b3704e0e43daa7538b469daac8880814e6a1e9a6197fc"}
Sep 10 23:19:49.405834 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.405816    2563 scope.go:117] "RemoveContainer" containerID="37e5be107009904f3f53c90a908ad120a6a87ffee2abcd4262af73607e2f8ba0"
Sep 10 23:19:49.406094 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:19:49.406075    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e5be107009904f3f53c90a908ad120a6a87ffee2abcd4262af73607e2f8ba0\": container with ID starting with 37e5be107009904f3f53c90a908ad120a6a87ffee2abcd4262af73607e2f8ba0 not found: ID does not exist" containerID="37e5be107009904f3f53c90a908ad120a6a87ffee2abcd4262af73607e2f8ba0"
Sep 10 23:19:49.406176 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.406105    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e5be107009904f3f53c90a908ad120a6a87ffee2abcd4262af73607e2f8ba0"} err="failed to get container status \"37e5be107009904f3f53c90a908ad120a6a87ffee2abcd4262af73607e2f8ba0\": rpc error: code = NotFound desc = could not find container \"37e5be107009904f3f53c90a908ad120a6a87ffee2abcd4262af73607e2f8ba0\": container with ID starting with 37e5be107009904f3f53c90a908ad120a6a87ffee2abcd4262af73607e2f8ba0 not found: ID does not exist"
Sep 10 23:19:49.406176 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.406127    2563 scope.go:117] "RemoveContainer" containerID="cec45fbd0ce6335de7ee93f5c50ebf511a5b049072b9a850dfe69bca08a4c907"
Sep 10 23:19:49.414453 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.414437    2563 scope.go:117] "RemoveContainer" containerID="cec45fbd0ce6335de7ee93f5c50ebf511a5b049072b9a850dfe69bca08a4c907"
Sep 10 23:19:49.414796 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:19:49.414775    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec45fbd0ce6335de7ee93f5c50ebf511a5b049072b9a850dfe69bca08a4c907\": container with ID starting with cec45fbd0ce6335de7ee93f5c50ebf511a5b049072b9a850dfe69bca08a4c907 not found: ID does not exist" containerID="cec45fbd0ce6335de7ee93f5c50ebf511a5b049072b9a850dfe69bca08a4c907"
Sep 10 23:19:49.414850 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.414803    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec45fbd0ce6335de7ee93f5c50ebf511a5b049072b9a850dfe69bca08a4c907"} err="failed to get container status \"cec45fbd0ce6335de7ee93f5c50ebf511a5b049072b9a850dfe69bca08a4c907\": rpc error: code = NotFound desc = could not find container \"cec45fbd0ce6335de7ee93f5c50ebf511a5b049072b9a850dfe69bca08a4c907\": container with ID starting with cec45fbd0ce6335de7ee93f5c50ebf511a5b049072b9a850dfe69bca08a4c907 not found: ID does not exist"
Sep 10 23:19:49.430992 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.430970    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"]
Sep 10 23:19:49.439422 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.439402    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b3d9be-59fbb6466-shg8n"]
Sep 10 23:19:49.460690 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.460668    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"]
Sep 10 23:19:49.462536 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.462517    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b3d9be-54c486c86f-9cbfx"]
Sep 10 23:19:49.814646 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.814619    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="862ad768-cdf7-4b2d-8601-02d43f4fdab6" path="/var/lib/kubelet/pods/862ad768-cdf7-4b2d-8601-02d43f4fdab6/volumes"
Sep 10 23:19:49.814996 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:49.814966    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3c7d380-0d07-4bd8-a461-357fa7e09964" path="/var/lib/kubelet/pods/c3c7d380-0d07-4bd8-a461-357fa7e09964/volumes"
Sep 10 23:19:54.068889 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:54.068850    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"]
Sep 10 23:19:59.050889 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.050854    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"]
Sep 10 23:19:59.051366 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.051334    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="862ad768-cdf7-4b2d-8601-02d43f4fdab6" containerName="payload-processing"
Sep 10 23:19:59.051366 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.051349    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="862ad768-cdf7-4b2d-8601-02d43f4fdab6" containerName="payload-processing"
Sep 10 23:19:59.051366 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.051363    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c7d380-0d07-4bd8-a461-357fa7e09964" containerName="payload-pre-processing"
Sep 10 23:19:59.051366 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.051369    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c7d380-0d07-4bd8-a461-357fa7e09964" containerName="payload-pre-processing"
Sep 10 23:19:59.051651 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.051437    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c7d380-0d07-4bd8-a461-357fa7e09964" containerName="payload-pre-processing"
Sep 10 23:19:59.051651 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.051446    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="862ad768-cdf7-4b2d-8601-02d43f4fdab6" containerName="payload-processing"
Sep 10 23:19:59.056322 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.056298    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.058759 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.058737    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce17e\"/\"distinct-w0-6ce17e-kserve-self-signed-certs\""
Sep 10 23:19:59.064082 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.064059    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"]
Sep 10 23:19:59.145959 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.145931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-tmp-dir\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.146139 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.145967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec235e1d-f185-45fa-9216-430298790d8c-tls-certs\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.146139 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.146012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-model-cache\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.146139 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.146068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-home\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.146139 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.146086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-dshm\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.146139 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.146127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6282m\" (UniqueName: \"kubernetes.io/projected/ec235e1d-f185-45fa-9216-430298790d8c-kube-api-access-6282m\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.246606 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.246575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6282m\" (UniqueName: \"kubernetes.io/projected/ec235e1d-f185-45fa-9216-430298790d8c-kube-api-access-6282m\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.246606 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.246617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-tmp-dir\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.246850 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.246643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec235e1d-f185-45fa-9216-430298790d8c-tls-certs\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.246850 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.246699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-model-cache\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.246850 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.246764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-home\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.246850 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.246790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-dshm\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.247134 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.247004    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-tmp-dir\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.247134 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.247116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-model-cache\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.247221 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.247142    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-home\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.248992 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.248974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-dshm\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.249229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.249213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec235e1d-f185-45fa-9216-430298790d8c-tls-certs\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.262619 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.262597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6282m\" (UniqueName: \"kubernetes.io/projected/ec235e1d-f185-45fa-9216-430298790d8c-kube-api-access-6282m\") pod \"distinct-w0-6ce17e-kserve-6788b999c4-t8xff\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") " pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.368263 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.368225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:19:59.639484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:19:59.639388    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"]
Sep 10 23:19:59.646720 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:19:59.646693    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec235e1d_f185_45fa_9216_430298790d8c.slice/crio-8956b0d9431930189f616ae2b075b00fed8f48e16033df2dc8c0ddf79a941c56 WatchSource:0}: Error finding container 8956b0d9431930189f616ae2b075b00fed8f48e16033df2dc8c0ddf79a941c56: Status 404 returned error can't find the container with id 8956b0d9431930189f616ae2b075b00fed8f48e16033df2dc8c0ddf79a941c56
Sep 10 23:20:00.465535 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:00.465502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff" event={"ID":"ec235e1d-f185-45fa-9216-430298790d8c","Type":"ContainerStarted","Data":"c96ddd403851911781891a9221ff86c8327c9291e17e20008fac9ce5c100cb67"}
Sep 10 23:20:00.465898 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:00.465540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff" event={"ID":"ec235e1d-f185-45fa-9216-430298790d8c","Type":"ContainerStarted","Data":"8956b0d9431930189f616ae2b075b00fed8f48e16033df2dc8c0ddf79a941c56"}
Sep 10 23:20:00.484014 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:00.483966    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff" podStartSLOduration=1.483952163 podStartE2EDuration="1.483952163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:00.482148851 +0000 UTC m=+1121.173726072" watchObservedRunningTime="2026-09-10 23:20:00.483952163 +0000 UTC m=+1121.175529347"
Sep 10 23:20:07.172507 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:07.172455    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"]
Sep 10 23:20:07.176528 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:07.176483    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"
Sep 10 23:20:07.179429 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:07.179403    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-7b202f\""
Sep 10 23:20:07.179603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:07.179584    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-7b202f-dockercfg-rlw8t\""
Sep 10 23:20:07.184910 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:07.184887    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"]
Sep 10 23:20:07.232052 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:07.232012    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmlpb\" (UniqueName: \"kubernetes.io/projected/5484738f-b75c-4604-99b6-90819dd32d96-kube-api-access-tmlpb\") pod \"payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f\" (UID: \"5484738f-b75c-4604-99b6-90819dd32d96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"
Sep 10 23:20:07.232223 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:07.232157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5484738f-b75c-4604-99b6-90819dd32d96-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f\" (UID: \"5484738f-b75c-4604-99b6-90819dd32d96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"
Sep 10 23:20:07.333111 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:07.333080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5484738f-b75c-4604-99b6-90819dd32d96-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f\" (UID: \"5484738f-b75c-4604-99b6-90819dd32d96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"
Sep 10 23:20:07.333270 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:07.333146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmlpb\" (UniqueName: \"kubernetes.io/projected/5484738f-b75c-4604-99b6-90819dd32d96-kube-api-access-tmlpb\") pod \"payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f\" (UID: \"5484738f-b75c-4604-99b6-90819dd32d96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"
Sep 10 23:20:07.333814 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:07.333794    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5484738f-b75c-4604-99b6-90819dd32d96-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f\" (UID: \"5484738f-b75c-4604-99b6-90819dd32d96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"
Sep 10 23:20:07.341774 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:07.341744    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmlpb\" (UniqueName: \"kubernetes.io/projected/5484738f-b75c-4604-99b6-90819dd32d96-kube-api-access-tmlpb\") pod \"payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f\" (UID: \"5484738f-b75c-4604-99b6-90819dd32d96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"
Sep 10 23:20:07.489592 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:07.489515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"
Sep 10 23:20:07.760714 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:07.760686    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"]
Sep 10 23:20:07.762376 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:20:07.762352    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5484738f_b75c_4604_99b6_90819dd32d96.slice/crio-65ef108ab441676428306736674c39340beb0ab1e674deb4695bdec0514c7b04 WatchSource:0}: Error finding container 65ef108ab441676428306736674c39340beb0ab1e674deb4695bdec0514c7b04: Status 404 returned error can't find the container with id 65ef108ab441676428306736674c39340beb0ab1e674deb4695bdec0514c7b04
Sep 10 23:20:08.511939 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:08.511897    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f" event={"ID":"5484738f-b75c-4604-99b6-90819dd32d96","Type":"ContainerStarted","Data":"804889ee87d19fadc4e5b23083da7a6c76fe99a0f2c52fbb45d408535da3e2fc"}
Sep 10 23:20:08.512354 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:08.511990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f" event={"ID":"5484738f-b75c-4604-99b6-90819dd32d96","Type":"ContainerStarted","Data":"65ef108ab441676428306736674c39340beb0ab1e674deb4695bdec0514c7b04"}
Sep 10 23:20:08.512354 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:08.512019    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"
Sep 10 23:20:08.530789 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:08.530737    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f" podStartSLOduration=1.5307224160000001 podStartE2EDuration="1.530722416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:08.529064386 +0000 UTC m=+1129.220641591" watchObservedRunningTime="2026-09-10 23:20:08.530722416 +0000 UTC m=+1129.222299600"
Sep 10 23:20:09.088346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.088280    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n" podUID="20df8786-fdda-490f-8971-5dbb554d9f29" containerName="main" containerID="cri-o://ccda55bd135a6d10d4b7be85faad76a8151f3ccbc1e72c43d291e34b6786a5b6" gracePeriod=45
Sep 10 23:20:09.340156 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.340098    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:20:09.369413 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.369370    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:20:09.369662 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.369457    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:20:09.388268 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.388246    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:20:09.452908 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.452879    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20df8786-fdda-490f-8971-5dbb554d9f29-tls-certs\") pod \"20df8786-fdda-490f-8971-5dbb554d9f29\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") "
Sep 10 23:20:09.453070 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.452934    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcjv2\" (UniqueName: \"kubernetes.io/projected/20df8786-fdda-490f-8971-5dbb554d9f29-kube-api-access-tcjv2\") pod \"20df8786-fdda-490f-8971-5dbb554d9f29\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") "
Sep 10 23:20:09.453070 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.453002    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-dshm\") pod \"20df8786-fdda-490f-8971-5dbb554d9f29\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") "
Sep 10 23:20:09.453070 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.453031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-tmp-dir\") pod \"20df8786-fdda-490f-8971-5dbb554d9f29\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") "
Sep 10 23:20:09.453070 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.453065    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-model-cache\") pod \"20df8786-fdda-490f-8971-5dbb554d9f29\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") "
Sep 10 23:20:09.453286 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.453126    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-home\") pod \"20df8786-fdda-490f-8971-5dbb554d9f29\" (UID: \"20df8786-fdda-490f-8971-5dbb554d9f29\") "
Sep 10 23:20:09.453682 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.453626    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-home" (OuterVolumeSpecName: "home") pod "20df8786-fdda-490f-8971-5dbb554d9f29" (UID: "20df8786-fdda-490f-8971-5dbb554d9f29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:20:09.454054 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.454028    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-model-cache" (OuterVolumeSpecName: "model-cache") pod "20df8786-fdda-490f-8971-5dbb554d9f29" (UID: "20df8786-fdda-490f-8971-5dbb554d9f29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:20:09.454114 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.454039    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20df8786-fdda-490f-8971-5dbb554d9f29" (UID: "20df8786-fdda-490f-8971-5dbb554d9f29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:20:09.455436 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.455417    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-dshm" (OuterVolumeSpecName: "dshm") pod "20df8786-fdda-490f-8971-5dbb554d9f29" (UID: "20df8786-fdda-490f-8971-5dbb554d9f29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:20:09.455620 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.455596    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20df8786-fdda-490f-8971-5dbb554d9f29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20df8786-fdda-490f-8971-5dbb554d9f29" (UID: "20df8786-fdda-490f-8971-5dbb554d9f29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:20:09.455620 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.455609    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20df8786-fdda-490f-8971-5dbb554d9f29-kube-api-access-tcjv2" (OuterVolumeSpecName: "kube-api-access-tcjv2") pod "20df8786-fdda-490f-8971-5dbb554d9f29" (UID: "20df8786-fdda-490f-8971-5dbb554d9f29"). InnerVolumeSpecName "kube-api-access-tcjv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:20:09.519705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.519669    2563 generic.go:356] "Generic (PLEG): container finished" podID="20df8786-fdda-490f-8971-5dbb554d9f29" containerID="ccda55bd135a6d10d4b7be85faad76a8151f3ccbc1e72c43d291e34b6786a5b6" exitCode=0
Sep 10 23:20:09.520136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.519723    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"
Sep 10 23:20:09.520136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.519760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n" event={"ID":"20df8786-fdda-490f-8971-5dbb554d9f29","Type":"ContainerDied","Data":"ccda55bd135a6d10d4b7be85faad76a8151f3ccbc1e72c43d291e34b6786a5b6"}
Sep 10 23:20:09.520136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.519809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n" event={"ID":"20df8786-fdda-490f-8971-5dbb554d9f29","Type":"ContainerDied","Data":"6f42e4c1c9aea513d46cc252f11bd5a94c40399796a152e91a90e31908ee1254"}
Sep 10 23:20:09.520136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.519833    2563 scope.go:117] "RemoveContainer" containerID="ccda55bd135a6d10d4b7be85faad76a8151f3ccbc1e72c43d291e34b6786a5b6"
Sep 10 23:20:09.530100 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.530076    2563 scope.go:117] "RemoveContainer" containerID="ccda55bd135a6d10d4b7be85faad76a8151f3ccbc1e72c43d291e34b6786a5b6"
Sep 10 23:20:09.530365 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:09.530345    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccda55bd135a6d10d4b7be85faad76a8151f3ccbc1e72c43d291e34b6786a5b6\": container with ID starting with ccda55bd135a6d10d4b7be85faad76a8151f3ccbc1e72c43d291e34b6786a5b6 not found: ID does not exist" containerID="ccda55bd135a6d10d4b7be85faad76a8151f3ccbc1e72c43d291e34b6786a5b6"
Sep 10 23:20:09.530447 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.530375    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccda55bd135a6d10d4b7be85faad76a8151f3ccbc1e72c43d291e34b6786a5b6"} err="failed to get container status \"ccda55bd135a6d10d4b7be85faad76a8151f3ccbc1e72c43d291e34b6786a5b6\": rpc error: code = NotFound desc = could not find container \"ccda55bd135a6d10d4b7be85faad76a8151f3ccbc1e72c43d291e34b6786a5b6\": container with ID starting with ccda55bd135a6d10d4b7be85faad76a8151f3ccbc1e72c43d291e34b6786a5b6 not found: ID does not exist"
Sep 10 23:20:09.532085 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.532069    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:20:09.542064 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.542041    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"]
Sep 10 23:20:09.545946 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.545927    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-b3d9be/explicit-model-b3d9be-kserve-86999d748c-6p54n"]
Sep 10 23:20:09.554784 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.554764    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:09.554784 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.554785    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20df8786-fdda-490f-8971-5dbb554d9f29-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:09.554906 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.554795    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcjv2\" (UniqueName: \"kubernetes.io/projected/20df8786-fdda-490f-8971-5dbb554d9f29-kube-api-access-tcjv2\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:09.554906 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.554803    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:09.554906 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.554810    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:09.554906 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.554818    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20df8786-fdda-490f-8971-5dbb554d9f29-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:09.816184 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:09.816154    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20df8786-fdda-490f-8971-5dbb554d9f29" path="/var/lib/kubelet/pods/20df8786-fdda-490f-8971-5dbb554d9f29/volumes"
Sep 10 23:20:19.522521 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:19.522480    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"
Sep 10 23:20:30.998801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:30.998762    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7843-7670a4"]
Sep 10 23:20:30.999264 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:30.999175    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20df8786-fdda-490f-8971-5dbb554d9f29" containerName="main"
Sep 10 23:20:30.999264 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:30.999186    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="20df8786-fdda-490f-8971-5dbb554d9f29" containerName="main"
Sep 10 23:20:30.999344 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:30.999269    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="20df8786-fdda-490f-8971-5dbb554d9f29" containerName="main"
Sep 10 23:20:31.003780 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:31.003763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-7670a4"
Sep 10 23:20:31.008122 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:31.008104    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f4p5h\""
Sep 10 23:20:31.012553 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:31.012533    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7843-7670a4"]
Sep 10 23:20:31.067439 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:31.067405    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2fzs\" (UniqueName: \"kubernetes.io/projected/727d6f82-3f1d-4bac-b667-e799e01199e5-kube-api-access-j2fzs\") pod \"test-curl-7843-7670a4\" (UID: \"727d6f82-3f1d-4bac-b667-e799e01199e5\") " pod="opendatahub/test-curl-7843-7670a4"
Sep 10 23:20:31.168055 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:31.168023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2fzs\" (UniqueName: \"kubernetes.io/projected/727d6f82-3f1d-4bac-b667-e799e01199e5-kube-api-access-j2fzs\") pod \"test-curl-7843-7670a4\" (UID: \"727d6f82-3f1d-4bac-b667-e799e01199e5\") " pod="opendatahub/test-curl-7843-7670a4"
Sep 10 23:20:31.175166 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:31.175139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2fzs\" (UniqueName: \"kubernetes.io/projected/727d6f82-3f1d-4bac-b667-e799e01199e5-kube-api-access-j2fzs\") pod \"test-curl-7843-7670a4\" (UID: \"727d6f82-3f1d-4bac-b667-e799e01199e5\") " pod="opendatahub/test-curl-7843-7670a4"
Sep 10 23:20:31.313957 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:31.313928    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-7670a4"
Sep 10 23:20:31.587921 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:31.587897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7843-7670a4"]
Sep 10 23:20:31.589855 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:20:31.589828    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod727d6f82_3f1d_4bac_b667_e799e01199e5.slice/crio-ecebe406380b77c670dd40250e1e41fe9a64c125ea3f6f528f3d03c9d1e64ff0 WatchSource:0}: Error finding container ecebe406380b77c670dd40250e1e41fe9a64c125ea3f6f528f3d03c9d1e64ff0: Status 404 returned error can't find the container with id ecebe406380b77c670dd40250e1e41fe9a64c125ea3f6f528f3d03c9d1e64ff0
Sep 10 23:20:31.648560 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:31.648532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7843-7670a4" event={"ID":"727d6f82-3f1d-4bac-b667-e799e01199e5","Type":"ContainerStarted","Data":"ecebe406380b77c670dd40250e1e41fe9a64c125ea3f6f528f3d03c9d1e64ff0"}
Sep 10 23:20:32.655878 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:32.655839    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7843-7670a4" event={"ID":"727d6f82-3f1d-4bac-b667-e799e01199e5","Type":"ContainerStarted","Data":"3c0bdfec307a1d57e759c6a64d5e31b46687efbc39f19e4838296838da610cf7"}
Sep 10 23:20:32.671342 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:32.671294    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7843-7670a4" podStartSLOduration=2.671276413 podStartE2EDuration="2.671276413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:32.668984988 +0000 UTC m=+1153.360562172" watchObservedRunningTime="2026-09-10 23:20:32.671276413 +0000 UTC m=+1153.362853597"
Sep 10 23:20:33.266885 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:33.266847    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7843-7670a4"]
Sep 10 23:20:33.270666 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:33.270636    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7843-7670a4"]
Sep 10 23:20:33.386210 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:33.386177    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7843-14d639"]
Sep 10 23:20:33.386621 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:33.386608    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727d6f82-3f1d-4bac-b667-e799e01199e5" containerName="test-curl-7843-7670a4"
Sep 10 23:20:33.386685 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:33.386623    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="727d6f82-3f1d-4bac-b667-e799e01199e5" containerName="test-curl-7843-7670a4"
Sep 10 23:20:33.389448 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:33.387391    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="727d6f82-3f1d-4bac-b667-e799e01199e5" containerName="test-curl-7843-7670a4"
Sep 10 23:20:33.395622 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:33.395290    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-14d639"
Sep 10 23:20:33.397359 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:33.397327    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7843-14d639"]
Sep 10 23:20:33.486988 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:33.486952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjxpm\" (UniqueName: \"kubernetes.io/projected/54d0f43e-20d3-4891-819e-54190bdb8725-kube-api-access-bjxpm\") pod \"test-curl-7843-14d639\" (UID: \"54d0f43e-20d3-4891-819e-54190bdb8725\") " pod="opendatahub/test-curl-7843-14d639"
Sep 10 23:20:33.587784 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:33.587750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjxpm\" (UniqueName: \"kubernetes.io/projected/54d0f43e-20d3-4891-819e-54190bdb8725-kube-api-access-bjxpm\") pod \"test-curl-7843-14d639\" (UID: \"54d0f43e-20d3-4891-819e-54190bdb8725\") " pod="opendatahub/test-curl-7843-14d639"
Sep 10 23:20:33.595627 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:33.595601    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjxpm\" (UniqueName: \"kubernetes.io/projected/54d0f43e-20d3-4891-819e-54190bdb8725-kube-api-access-bjxpm\") pod \"test-curl-7843-14d639\" (UID: \"54d0f43e-20d3-4891-819e-54190bdb8725\") " pod="opendatahub/test-curl-7843-14d639"
Sep 10 23:20:33.661220 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:33.661149    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7843-7670a4" container="test-curl-7843-7670a4"
Sep 10 23:20:33.710222 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:33.710196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-14d639"
Sep 10 23:20:33.985327 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:33.985303    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7843-14d639"]
Sep 10 23:20:33.987182 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:20:33.987155    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54d0f43e_20d3_4891_819e_54190bdb8725.slice/crio-38ffe8cf5a1f82b538daae6f88f595c9259a6b84530d147303e134e8ec914a87 WatchSource:0}: Error finding container 38ffe8cf5a1f82b538daae6f88f595c9259a6b84530d147303e134e8ec914a87: Status 404 returned error can't find the container with id 38ffe8cf5a1f82b538daae6f88f595c9259a6b84530d147303e134e8ec914a87
Sep 10 23:20:34.669352 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:34.669301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7843-14d639" event={"ID":"54d0f43e-20d3-4891-819e-54190bdb8725","Type":"ContainerStarted","Data":"6564b5554e340b7ef3e139a8c7f9a0843dcaf01fd31a9a79a43695c3bd8d231b"}
Sep 10 23:20:34.669926 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:34.669353    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7843-7670a4" podUID="727d6f82-3f1d-4bac-b667-e799e01199e5" containerName="test-curl-7843-7670a4" containerID="cri-o://3c0bdfec307a1d57e759c6a64d5e31b46687efbc39f19e4838296838da610cf7" gracePeriod=2
Sep 10 23:20:34.669926 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:34.669372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7843-14d639" event={"ID":"54d0f43e-20d3-4891-819e-54190bdb8725","Type":"ContainerStarted","Data":"38ffe8cf5a1f82b538daae6f88f595c9259a6b84530d147303e134e8ec914a87"}
Sep 10 23:20:34.685695 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:34.685644    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7843-14d639" podStartSLOduration=1.685628387 podStartE2EDuration="1.685628387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:34.68504702 +0000 UTC m=+1155.376624204" watchObservedRunningTime="2026-09-10 23:20:34.685628387 +0000 UTC m=+1155.377205573"
Sep 10 23:20:35.262923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:35.262884    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7843-14d639"]
Sep 10 23:20:35.269442 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:35.269410    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7843-14d639"]
Sep 10 23:20:35.374763 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:35.374727    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7843-b86a62"]
Sep 10 23:20:35.375235 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:35.375216    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d0f43e-20d3-4891-819e-54190bdb8725" containerName="test-curl-7843-14d639"
Sep 10 23:20:35.375325 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:35.375236    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d0f43e-20d3-4891-819e-54190bdb8725" containerName="test-curl-7843-14d639"
Sep 10 23:20:35.375402 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:35.375389    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="54d0f43e-20d3-4891-819e-54190bdb8725" containerName="test-curl-7843-14d639"
Sep 10 23:20:35.378759 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:35.378741    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-b86a62"
Sep 10 23:20:35.383439 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:35.383420    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7843-b86a62"]
Sep 10 23:20:35.405234 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:35.405192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsczx\" (UniqueName: \"kubernetes.io/projected/1aa9692c-9743-4ff5-b3e5-86dfc012bb17-kube-api-access-hsczx\") pod \"test-curl-7843-b86a62\" (UID: \"1aa9692c-9743-4ff5-b3e5-86dfc012bb17\") " pod="opendatahub/test-curl-7843-b86a62"
Sep 10 23:20:35.506763 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:35.506714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsczx\" (UniqueName: \"kubernetes.io/projected/1aa9692c-9743-4ff5-b3e5-86dfc012bb17-kube-api-access-hsczx\") pod \"test-curl-7843-b86a62\" (UID: \"1aa9692c-9743-4ff5-b3e5-86dfc012bb17\") " pod="opendatahub/test-curl-7843-b86a62"
Sep 10 23:20:35.514523 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:35.514473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsczx\" (UniqueName: \"kubernetes.io/projected/1aa9692c-9743-4ff5-b3e5-86dfc012bb17-kube-api-access-hsczx\") pod \"test-curl-7843-b86a62\" (UID: \"1aa9692c-9743-4ff5-b3e5-86dfc012bb17\") " pod="opendatahub/test-curl-7843-b86a62"
Sep 10 23:20:35.674981 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:35.674891    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7843-14d639" container="test-curl-7843-14d639"
Sep 10 23:20:35.689718 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:35.689694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-b86a62"
Sep 10 23:20:35.966934 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:35.966908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7843-b86a62"]
Sep 10 23:20:35.968467 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:20:35.968440    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1aa9692c_9743_4ff5_b3e5_86dfc012bb17.slice/crio-166724a6645dc4e07f59b1805a72d018166b406e769a6d207907a049c1e82378 WatchSource:0}: Error finding container 166724a6645dc4e07f59b1805a72d018166b406e769a6d207907a049c1e82378: Status 404 returned error can't find the container with id 166724a6645dc4e07f59b1805a72d018166b406e769a6d207907a049c1e82378
Sep 10 23:20:36.681763 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:36.681735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7843-b86a62" event={"ID":"1aa9692c-9743-4ff5-b3e5-86dfc012bb17","Type":"ContainerStarted","Data":"ea218d48938c8319044b14e3f19aa29d298877bb0cb029faeebfbe775dee7bf1"}
Sep 10 23:20:36.682101 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:36.681779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7843-b86a62" event={"ID":"1aa9692c-9743-4ff5-b3e5-86dfc012bb17","Type":"ContainerStarted","Data":"166724a6645dc4e07f59b1805a72d018166b406e769a6d207907a049c1e82378"}
Sep 10 23:20:36.682101 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:36.681896    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7843-14d639" podUID="54d0f43e-20d3-4891-819e-54190bdb8725" containerName="test-curl-7843-14d639" containerID="cri-o://6564b5554e340b7ef3e139a8c7f9a0843dcaf01fd31a9a79a43695c3bd8d231b" gracePeriod=2
Sep 10 23:20:36.697597 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:36.697550    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7843-b86a62" podStartSLOduration=1.697533687 podStartE2EDuration="1.697533687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:36.696017182 +0000 UTC m=+1157.387594369" watchObservedRunningTime="2026-09-10 23:20:36.697533687 +0000 UTC m=+1157.389110871"
Sep 10 23:20:36.850212 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:36.850191    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-7670a4"
Sep 10 23:20:36.921752 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:36.921722    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2fzs\" (UniqueName: \"kubernetes.io/projected/727d6f82-3f1d-4bac-b667-e799e01199e5-kube-api-access-j2fzs\") pod \"727d6f82-3f1d-4bac-b667-e799e01199e5\" (UID: \"727d6f82-3f1d-4bac-b667-e799e01199e5\") "
Sep 10 23:20:36.923708 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:36.923651    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727d6f82-3f1d-4bac-b667-e799e01199e5-kube-api-access-j2fzs" (OuterVolumeSpecName: "kube-api-access-j2fzs") pod "727d6f82-3f1d-4bac-b667-e799e01199e5" (UID: "727d6f82-3f1d-4bac-b667-e799e01199e5"). InnerVolumeSpecName "kube-api-access-j2fzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:20:37.022766 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.022737    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2fzs\" (UniqueName: \"kubernetes.io/projected/727d6f82-3f1d-4bac-b667-e799e01199e5-kube-api-access-j2fzs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:37.255118 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.255040    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7843-b86a62"]
Sep 10 23:20:37.260713 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.260688    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7843-b86a62"]
Sep 10 23:20:37.373921 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.373890    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7843-962379"]
Sep 10 23:20:37.374317 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.374302    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aa9692c-9743-4ff5-b3e5-86dfc012bb17" containerName="test-curl-7843-b86a62"
Sep 10 23:20:37.374366 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.374319    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aa9692c-9743-4ff5-b3e5-86dfc012bb17" containerName="test-curl-7843-b86a62"
Sep 10 23:20:37.375149 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.375118    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aa9692c-9743-4ff5-b3e5-86dfc012bb17" containerName="test-curl-7843-b86a62"
Sep 10 23:20:37.379713 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.379692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-962379"
Sep 10 23:20:37.386474 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.386445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7843-962379"]
Sep 10 23:20:37.427531 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.427471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27ls8\" (UniqueName: \"kubernetes.io/projected/0480c164-13d1-462f-9b2a-90fc080de8ed-kube-api-access-27ls8\") pod \"test-curl-7843-962379\" (UID: \"0480c164-13d1-462f-9b2a-90fc080de8ed\") " pod="opendatahub/test-curl-7843-962379"
Sep 10 23:20:37.528146 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.528061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27ls8\" (UniqueName: \"kubernetes.io/projected/0480c164-13d1-462f-9b2a-90fc080de8ed-kube-api-access-27ls8\") pod \"test-curl-7843-962379\" (UID: \"0480c164-13d1-462f-9b2a-90fc080de8ed\") " pod="opendatahub/test-curl-7843-962379"
Sep 10 23:20:37.535601 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.535569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27ls8\" (UniqueName: \"kubernetes.io/projected/0480c164-13d1-462f-9b2a-90fc080de8ed-kube-api-access-27ls8\") pod \"test-curl-7843-962379\" (UID: \"0480c164-13d1-462f-9b2a-90fc080de8ed\") " pod="opendatahub/test-curl-7843-962379"
Sep 10 23:20:37.688623 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.688578    2563 generic.go:356] "Generic (PLEG): container finished" podID="727d6f82-3f1d-4bac-b667-e799e01199e5" containerID="3c0bdfec307a1d57e759c6a64d5e31b46687efbc39f19e4838296838da610cf7" exitCode=137
Sep 10 23:20:37.689056 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.688642    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-7670a4"
Sep 10 23:20:37.689056 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.688678    2563 scope.go:117] "RemoveContainer" containerID="3c0bdfec307a1d57e759c6a64d5e31b46687efbc39f19e4838296838da610cf7"
Sep 10 23:20:37.689056 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:37.688965    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7843-b86a62" container="test-curl-7843-b86a62"
Sep 10 23:20:37.691898 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.691865    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-962379"
Sep 10 23:20:37.699578 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.699556    2563 scope.go:117] "RemoveContainer" containerID="3c0bdfec307a1d57e759c6a64d5e31b46687efbc39f19e4838296838da610cf7"
Sep 10 23:20:37.699742 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:37.699709    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7843-7670a4\" is forbidden: User \"system:node:ip-10-0-130-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-246.ec2.internal' and this object" podUID="727d6f82-3f1d-4bac-b667-e799e01199e5" pod="opendatahub/test-curl-7843-7670a4"
Sep 10 23:20:37.699901 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:37.699881    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0bdfec307a1d57e759c6a64d5e31b46687efbc39f19e4838296838da610cf7\": container with ID starting with 3c0bdfec307a1d57e759c6a64d5e31b46687efbc39f19e4838296838da610cf7 not found: ID does not exist" containerID="3c0bdfec307a1d57e759c6a64d5e31b46687efbc39f19e4838296838da610cf7"
Sep 10 23:20:37.699982 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.699909    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0bdfec307a1d57e759c6a64d5e31b46687efbc39f19e4838296838da610cf7"} err="failed to get container status \"3c0bdfec307a1d57e759c6a64d5e31b46687efbc39f19e4838296838da610cf7\": rpc error: code = NotFound desc = could not find container \"3c0bdfec307a1d57e759c6a64d5e31b46687efbc39f19e4838296838da610cf7\": container with ID starting with 3c0bdfec307a1d57e759c6a64d5e31b46687efbc39f19e4838296838da610cf7 not found: ID does not exist"
Sep 10 23:20:37.816121 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.816091    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="727d6f82-3f1d-4bac-b667-e799e01199e5" path="/var/lib/kubelet/pods/727d6f82-3f1d-4bac-b667-e799e01199e5/volumes"
Sep 10 23:20:37.999262 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:37.999239    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7843-962379"]
Sep 10 23:20:38.001302 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:20:38.001268    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0480c164_13d1_462f_9b2a_90fc080de8ed.slice/crio-658c4543adc64cd48730770487e691d5fc293e74f17d5359c2797a092e8268b3 WatchSource:0}: Error finding container 658c4543adc64cd48730770487e691d5fc293e74f17d5359c2797a092e8268b3: Status 404 returned error can't find the container with id 658c4543adc64cd48730770487e691d5fc293e74f17d5359c2797a092e8268b3
Sep 10 23:20:38.697456 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:38.697243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7843-962379" event={"ID":"0480c164-13d1-462f-9b2a-90fc080de8ed","Type":"ContainerStarted","Data":"41a4ff504ae55b7aaf26a26855957055367430451f02f212aa91abed45b96671"}
Sep 10 23:20:38.697456 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:38.697294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7843-962379" event={"ID":"0480c164-13d1-462f-9b2a-90fc080de8ed","Type":"ContainerStarted","Data":"658c4543adc64cd48730770487e691d5fc293e74f17d5359c2797a092e8268b3"}
Sep 10 23:20:38.698344 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:38.698320    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7843-b86a62" podUID="1aa9692c-9743-4ff5-b3e5-86dfc012bb17" containerName="test-curl-7843-b86a62" containerID="cri-o://ea218d48938c8319044b14e3f19aa29d298877bb0cb029faeebfbe775dee7bf1" gracePeriod=2
Sep 10 23:20:38.711099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:38.711059    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7843-962379" podStartSLOduration=1.711044733 podStartE2EDuration="1.711044733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:38.710178955 +0000 UTC m=+1159.401756151" watchObservedRunningTime="2026-09-10 23:20:38.711044733 +0000 UTC m=+1159.402621914"
Sep 10 23:20:38.870366 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:38.870343    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-14d639"
Sep 10 23:20:38.940612 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:38.940549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjxpm\" (UniqueName: \"kubernetes.io/projected/54d0f43e-20d3-4891-819e-54190bdb8725-kube-api-access-bjxpm\") pod \"54d0f43e-20d3-4891-819e-54190bdb8725\" (UID: \"54d0f43e-20d3-4891-819e-54190bdb8725\") "
Sep 10 23:20:38.942348 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:38.942325    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d0f43e-20d3-4891-819e-54190bdb8725-kube-api-access-bjxpm" (OuterVolumeSpecName: "kube-api-access-bjxpm") pod "54d0f43e-20d3-4891-819e-54190bdb8725" (UID: "54d0f43e-20d3-4891-819e-54190bdb8725"). InnerVolumeSpecName "kube-api-access-bjxpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:20:39.042130 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:39.042095    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjxpm\" (UniqueName: \"kubernetes.io/projected/54d0f43e-20d3-4891-819e-54190bdb8725-kube-api-access-bjxpm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:39.271403 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:39.271314    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7843-962379"]
Sep 10 23:20:39.275790 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:39.275764    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7843-962379"]
Sep 10 23:20:39.704863 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:39.704833    2563 generic.go:356] "Generic (PLEG): container finished" podID="54d0f43e-20d3-4891-819e-54190bdb8725" containerID="6564b5554e340b7ef3e139a8c7f9a0843dcaf01fd31a9a79a43695c3bd8d231b" exitCode=137
Sep 10 23:20:39.705313 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:39.704875    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-14d639"
Sep 10 23:20:39.705313 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:39.704945    2563 scope.go:117] "RemoveContainer" containerID="6564b5554e340b7ef3e139a8c7f9a0843dcaf01fd31a9a79a43695c3bd8d231b"
Sep 10 23:20:39.705313 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:39.705089    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7843-962379" container="test-curl-7843-962379"
Sep 10 23:20:39.714657 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:39.714637    2563 scope.go:117] "RemoveContainer" containerID="6564b5554e340b7ef3e139a8c7f9a0843dcaf01fd31a9a79a43695c3bd8d231b"
Sep 10 23:20:39.714918 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:39.714899    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6564b5554e340b7ef3e139a8c7f9a0843dcaf01fd31a9a79a43695c3bd8d231b\": container with ID starting with 6564b5554e340b7ef3e139a8c7f9a0843dcaf01fd31a9a79a43695c3bd8d231b not found: ID does not exist" containerID="6564b5554e340b7ef3e139a8c7f9a0843dcaf01fd31a9a79a43695c3bd8d231b"
Sep 10 23:20:39.714965 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:39.714926    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6564b5554e340b7ef3e139a8c7f9a0843dcaf01fd31a9a79a43695c3bd8d231b"} err="failed to get container status \"6564b5554e340b7ef3e139a8c7f9a0843dcaf01fd31a9a79a43695c3bd8d231b\": rpc error: code = NotFound desc = could not find container \"6564b5554e340b7ef3e139a8c7f9a0843dcaf01fd31a9a79a43695c3bd8d231b\": container with ID starting with 6564b5554e340b7ef3e139a8c7f9a0843dcaf01fd31a9a79a43695c3bd8d231b not found: ID does not exist"
Sep 10 23:20:39.715283 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:39.715267    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7843-14d639\" is forbidden: User \"system:node:ip-10-0-130-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-246.ec2.internal' and this object" podUID="54d0f43e-20d3-4891-819e-54190bdb8725" pod="opendatahub/test-curl-7843-14d639"
Sep 10 23:20:39.812226 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:39.812192    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7843-14d639\" is forbidden: User \"system:node:ip-10-0-130-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-246.ec2.internal' and this object" podUID="54d0f43e-20d3-4891-819e-54190bdb8725" pod="opendatahub/test-curl-7843-14d639"
Sep 10 23:20:39.814455 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:39.814432    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54d0f43e-20d3-4891-819e-54190bdb8725" path="/var/lib/kubelet/pods/54d0f43e-20d3-4891-819e-54190bdb8725/volumes"
Sep 10 23:20:40.712386 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:40.712351    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7843-962379" podUID="0480c164-13d1-462f-9b2a-90fc080de8ed" containerName="test-curl-7843-962379" containerID="cri-o://41a4ff504ae55b7aaf26a26855957055367430451f02f212aa91abed45b96671" gracePeriod=2
Sep 10 23:20:40.896442 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:40.896419    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-b86a62"
Sep 10 23:20:40.964204 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:40.964149    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsczx\" (UniqueName: \"kubernetes.io/projected/1aa9692c-9743-4ff5-b3e5-86dfc012bb17-kube-api-access-hsczx\") pod \"1aa9692c-9743-4ff5-b3e5-86dfc012bb17\" (UID: \"1aa9692c-9743-4ff5-b3e5-86dfc012bb17\") "
Sep 10 23:20:40.966013 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:40.965984    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa9692c-9743-4ff5-b3e5-86dfc012bb17-kube-api-access-hsczx" (OuterVolumeSpecName: "kube-api-access-hsczx") pod "1aa9692c-9743-4ff5-b3e5-86dfc012bb17" (UID: "1aa9692c-9743-4ff5-b3e5-86dfc012bb17"). InnerVolumeSpecName "kube-api-access-hsczx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:20:41.065594 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:41.065552    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsczx\" (UniqueName: \"kubernetes.io/projected/1aa9692c-9743-4ff5-b3e5-86dfc012bb17-kube-api-access-hsczx\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:41.718394 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:41.718360    2563 generic.go:356] "Generic (PLEG): container finished" podID="1aa9692c-9743-4ff5-b3e5-86dfc012bb17" containerID="ea218d48938c8319044b14e3f19aa29d298877bb0cb029faeebfbe775dee7bf1" exitCode=137
Sep 10 23:20:41.718394 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:41.718398    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-b86a62"
Sep 10 23:20:41.718822 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:41.718430    2563 scope.go:117] "RemoveContainer" containerID="ea218d48938c8319044b14e3f19aa29d298877bb0cb029faeebfbe775dee7bf1"
Sep 10 23:20:41.727304 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:41.727285    2563 scope.go:117] "RemoveContainer" containerID="ea218d48938c8319044b14e3f19aa29d298877bb0cb029faeebfbe775dee7bf1"
Sep 10 23:20:41.727636 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:41.727613    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea218d48938c8319044b14e3f19aa29d298877bb0cb029faeebfbe775dee7bf1\": container with ID starting with ea218d48938c8319044b14e3f19aa29d298877bb0cb029faeebfbe775dee7bf1 not found: ID does not exist" containerID="ea218d48938c8319044b14e3f19aa29d298877bb0cb029faeebfbe775dee7bf1"
Sep 10 23:20:41.727718 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:41.727642    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea218d48938c8319044b14e3f19aa29d298877bb0cb029faeebfbe775dee7bf1"} err="failed to get container status \"ea218d48938c8319044b14e3f19aa29d298877bb0cb029faeebfbe775dee7bf1\": rpc error: code = NotFound desc = could not find container \"ea218d48938c8319044b14e3f19aa29d298877bb0cb029faeebfbe775dee7bf1\": container with ID starting with ea218d48938c8319044b14e3f19aa29d298877bb0cb029faeebfbe775dee7bf1 not found: ID does not exist"
Sep 10 23:20:41.728606 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:41.728581    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7843-b86a62\" is forbidden: User \"system:node:ip-10-0-130-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-246.ec2.internal' and this object" podUID="1aa9692c-9743-4ff5-b3e5-86dfc012bb17" pod="opendatahub/test-curl-7843-b86a62"
Sep 10 23:20:41.814814 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:41.814784    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aa9692c-9743-4ff5-b3e5-86dfc012bb17" path="/var/lib/kubelet/pods/1aa9692c-9743-4ff5-b3e5-86dfc012bb17/volumes"
Sep 10 23:20:42.898794 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:42.898763    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-962379"
Sep 10 23:20:42.984849 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:42.984771    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27ls8\" (UniqueName: \"kubernetes.io/projected/0480c164-13d1-462f-9b2a-90fc080de8ed-kube-api-access-27ls8\") pod \"0480c164-13d1-462f-9b2a-90fc080de8ed\" (UID: \"0480c164-13d1-462f-9b2a-90fc080de8ed\") "
Sep 10 23:20:42.986810 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:42.986781    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0480c164-13d1-462f-9b2a-90fc080de8ed-kube-api-access-27ls8" (OuterVolumeSpecName: "kube-api-access-27ls8") pod "0480c164-13d1-462f-9b2a-90fc080de8ed" (UID: "0480c164-13d1-462f-9b2a-90fc080de8ed"). InnerVolumeSpecName "kube-api-access-27ls8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:20:43.086408 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.086377    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27ls8\" (UniqueName: \"kubernetes.io/projected/0480c164-13d1-462f-9b2a-90fc080de8ed-kube-api-access-27ls8\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:43.461056 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.461025    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2"]
Sep 10 23:20:43.461466 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.461454    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0480c164-13d1-462f-9b2a-90fc080de8ed" containerName="test-curl-7843-962379"
Sep 10 23:20:43.461520 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.461469    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0480c164-13d1-462f-9b2a-90fc080de8ed" containerName="test-curl-7843-962379"
Sep 10 23:20:43.461569 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.461559    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0480c164-13d1-462f-9b2a-90fc080de8ed" containerName="test-curl-7843-962379"
Sep 10 23:20:43.464664 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.464647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2"
Sep 10 23:20:43.466741 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.466711    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:20:43.466837 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.466759    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:20:43.471453 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.471427    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2"]
Sep 10 23:20:43.591285 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.591253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aebe214-3cbb-4de9-a43a-0ae9e1beba10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-679c94-btmb2\" (UID: \"9aebe214-3cbb-4de9-a43a-0ae9e1beba10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2"
Sep 10 23:20:43.691974 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.691945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aebe214-3cbb-4de9-a43a-0ae9e1beba10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-679c94-btmb2\" (UID: \"9aebe214-3cbb-4de9-a43a-0ae9e1beba10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2"
Sep 10 23:20:43.692604 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.692578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aebe214-3cbb-4de9-a43a-0ae9e1beba10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-679c94-btmb2\" (UID: \"9aebe214-3cbb-4de9-a43a-0ae9e1beba10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2"
Sep 10 23:20:43.732307 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.732223    2563 generic.go:356] "Generic (PLEG): container finished" podID="0480c164-13d1-462f-9b2a-90fc080de8ed" containerID="41a4ff504ae55b7aaf26a26855957055367430451f02f212aa91abed45b96671" exitCode=137
Sep 10 23:20:43.732307 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.732261    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7843-962379"
Sep 10 23:20:43.732531 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.732331    2563 scope.go:117] "RemoveContainer" containerID="41a4ff504ae55b7aaf26a26855957055367430451f02f212aa91abed45b96671"
Sep 10 23:20:43.741815 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.741800    2563 scope.go:117] "RemoveContainer" containerID="41a4ff504ae55b7aaf26a26855957055367430451f02f212aa91abed45b96671"
Sep 10 23:20:43.742059 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:43.742042    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a4ff504ae55b7aaf26a26855957055367430451f02f212aa91abed45b96671\": container with ID starting with 41a4ff504ae55b7aaf26a26855957055367430451f02f212aa91abed45b96671 not found: ID does not exist" containerID="41a4ff504ae55b7aaf26a26855957055367430451f02f212aa91abed45b96671"
Sep 10 23:20:43.742100 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.742069    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a4ff504ae55b7aaf26a26855957055367430451f02f212aa91abed45b96671"} err="failed to get container status \"41a4ff504ae55b7aaf26a26855957055367430451f02f212aa91abed45b96671\": rpc error: code = NotFound desc = could not find container \"41a4ff504ae55b7aaf26a26855957055367430451f02f212aa91abed45b96671\": container with ID starting with 41a4ff504ae55b7aaf26a26855957055367430451f02f212aa91abed45b96671 not found: ID does not exist"
Sep 10 23:20:43.742379 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:43.742358    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7843-962379\" is forbidden: User \"system:node:ip-10-0-130-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-246.ec2.internal' and this object" podUID="0480c164-13d1-462f-9b2a-90fc080de8ed" pod="opendatahub/test-curl-7843-962379"
Sep 10 23:20:43.776682 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.776658    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2"
Sep 10 23:20:43.814777 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:43.814750    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0480c164-13d1-462f-9b2a-90fc080de8ed" path="/var/lib/kubelet/pods/0480c164-13d1-462f-9b2a-90fc080de8ed/volumes"
Sep 10 23:20:44.049096 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:44.049072    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2"]
Sep 10 23:20:44.051373 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:20:44.051344    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aebe214_3cbb_4de9_a43a_0ae9e1beba10.slice/crio-f52fef04b84699a80629f51aed16262f9089386b459cfe2753d1cfffb9ae88fd WatchSource:0}: Error finding container f52fef04b84699a80629f51aed16262f9089386b459cfe2753d1cfffb9ae88fd: Status 404 returned error can't find the container with id f52fef04b84699a80629f51aed16262f9089386b459cfe2753d1cfffb9ae88fd
Sep 10 23:20:44.741351 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:44.741271    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2" event={"ID":"9aebe214-3cbb-4de9-a43a-0ae9e1beba10","Type":"ContainerStarted","Data":"be26dea0588c1a9aafdf46a61051cf79fe3a60d1d45b36b9c085c4c50f5a4585"}
Sep 10 23:20:44.741351 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:44.741330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2" event={"ID":"9aebe214-3cbb-4de9-a43a-0ae9e1beba10","Type":"ContainerStarted","Data":"f52fef04b84699a80629f51aed16262f9089386b459cfe2753d1cfffb9ae88fd"}
Sep 10 23:20:44.757717 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:44.757674    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2" podStartSLOduration=1.7576605060000001 podStartE2EDuration="1.757660506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:44.755928525 +0000 UTC m=+1165.447505708" watchObservedRunningTime="2026-09-10 23:20:44.757660506 +0000 UTC m=+1165.449237689"
Sep 10 23:20:45.552455 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:45.552413    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"]
Sep 10 23:20:45.556214 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:45.556190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"
Sep 10 23:20:45.558641 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:45.558616    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-76145153-dockercfg-45626\""
Sep 10 23:20:45.558736 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:45.558692    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-76145153\""
Sep 10 23:20:45.565849 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:45.565824    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"]
Sep 10 23:20:45.611683 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:45.611646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39bffac5-509f-4094-9091-4b48041bd4f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd\" (UID: \"39bffac5-509f-4094-9091-4b48041bd4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"
Sep 10 23:20:45.611823 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:45.611705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlq2q\" (UniqueName: \"kubernetes.io/projected/39bffac5-509f-4094-9091-4b48041bd4f7-kube-api-access-vlq2q\") pod \"payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd\" (UID: \"39bffac5-509f-4094-9091-4b48041bd4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"
Sep 10 23:20:45.712669 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:45.712632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39bffac5-509f-4094-9091-4b48041bd4f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd\" (UID: \"39bffac5-509f-4094-9091-4b48041bd4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"
Sep 10 23:20:45.712669 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:45.712674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlq2q\" (UniqueName: \"kubernetes.io/projected/39bffac5-509f-4094-9091-4b48041bd4f7-kube-api-access-vlq2q\") pod \"payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd\" (UID: \"39bffac5-509f-4094-9091-4b48041bd4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"
Sep 10 23:20:45.713381 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:45.713358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39bffac5-509f-4094-9091-4b48041bd4f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd\" (UID: \"39bffac5-509f-4094-9091-4b48041bd4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"
Sep 10 23:20:45.720102 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:45.720069    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlq2q\" (UniqueName: \"kubernetes.io/projected/39bffac5-509f-4094-9091-4b48041bd4f7-kube-api-access-vlq2q\") pod \"payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd\" (UID: \"39bffac5-509f-4094-9091-4b48041bd4f7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"
Sep 10 23:20:45.749480 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:45.749448    2563 generic.go:356] "Generic (PLEG): container finished" podID="9aebe214-3cbb-4de9-a43a-0ae9e1beba10" containerID="be26dea0588c1a9aafdf46a61051cf79fe3a60d1d45b36b9c085c4c50f5a4585" exitCode=0
Sep 10 23:20:45.749613 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:45.749536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2" event={"ID":"9aebe214-3cbb-4de9-a43a-0ae9e1beba10","Type":"ContainerDied","Data":"be26dea0588c1a9aafdf46a61051cf79fe3a60d1d45b36b9c085c4c50f5a4585"}
Sep 10 23:20:45.867282 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:45.867248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"
Sep 10 23:20:46.139639 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:46.139614    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"]
Sep 10 23:20:46.141428 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:20:46.141405    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bffac5_509f_4094_9091_4b48041bd4f7.slice/crio-0b39b157f85b3e492afe4f57ef17bba2fcdc86f57091766a6820d618303a335e WatchSource:0}: Error finding container 0b39b157f85b3e492afe4f57ef17bba2fcdc86f57091766a6820d618303a335e: Status 404 returned error can't find the container with id 0b39b157f85b3e492afe4f57ef17bba2fcdc86f57091766a6820d618303a335e
Sep 10 23:20:46.757293 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:46.757255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd" event={"ID":"39bffac5-509f-4094-9091-4b48041bd4f7","Type":"ContainerStarted","Data":"f723f7ae3f19eb3febc5d02bd6551c901891564303f75aa86caa52998c324df4"}
Sep 10 23:20:46.757767 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:46.757357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"
Sep 10 23:20:46.757767 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:46.757389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd" event={"ID":"39bffac5-509f-4094-9091-4b48041bd4f7","Type":"ContainerStarted","Data":"0b39b157f85b3e492afe4f57ef17bba2fcdc86f57091766a6820d618303a335e"}
Sep 10 23:20:46.777214 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:46.777166    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd" podStartSLOduration=1.7771498879999998 podStartE2EDuration="1.777149888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:20:46.775583182 +0000 UTC m=+1167.467160391" watchObservedRunningTime="2026-09-10 23:20:46.777149888 +0000 UTC m=+1167.468727072"
Sep 10 23:20:46.898871 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:46.898847    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2"
Sep 10 23:20:47.026089 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:47.026014    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aebe214-3cbb-4de9-a43a-0ae9e1beba10-maas-api-service-ca\") pod \"9aebe214-3cbb-4de9-a43a-0ae9e1beba10\" (UID: \"9aebe214-3cbb-4de9-a43a-0ae9e1beba10\") "
Sep 10 23:20:47.026416 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:47.026390    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aebe214-3cbb-4de9-a43a-0ae9e1beba10-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9aebe214-3cbb-4de9-a43a-0ae9e1beba10" (UID: "9aebe214-3cbb-4de9-a43a-0ae9e1beba10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:20:47.127772 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:47.127740    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9aebe214-3cbb-4de9-a43a-0ae9e1beba10-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:47.763929 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:47.763899    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2"
Sep 10 23:20:47.764326 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:47.763936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2" event={"ID":"9aebe214-3cbb-4de9-a43a-0ae9e1beba10","Type":"ContainerDied","Data":"f52fef04b84699a80629f51aed16262f9089386b459cfe2753d1cfffb9ae88fd"}
Sep 10 23:20:47.764326 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:47.763981    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f52fef04b84699a80629f51aed16262f9089386b459cfe2753d1cfffb9ae88fd"
Sep 10 23:20:54.203219 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.203185    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"]
Sep 10 23:20:54.203709 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.203625    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl" podUID="ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a" containerName="payload-processing" containerID="cri-o://f2d8719ab9e923b032a694f6709f87eae5b3b4d9edd51fe60d476089d052b9d6" gracePeriod=30
Sep 10 23:20:54.213382 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.213357    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"]
Sep 10 23:20:54.213628 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.213604    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw" podUID="8f8835f2-b94d-44ab-8283-d4e750300326" containerName="payload-pre-processing" containerID="cri-o://99deed1b1635aa5be34b5c5cd52b4f6cf9363bfa5613c900b5a4129796499ad2" gracePeriod=30
Sep 10 23:20:54.358874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.358800    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2"]
Sep 10 23:20:54.361065 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.361031    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-679c94-btmb2"]
Sep 10 23:20:54.468991 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.468952    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"
Sep 10 23:20:54.484825 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.484801    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"
Sep 10 23:20:54.600506 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.600475    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrr44\" (UniqueName: \"kubernetes.io/projected/ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a-kube-api-access-mrr44\") pod \"ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a\" (UID: \"ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a\") "
Sep 10 23:20:54.600646 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.600535    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8835f2-b94d-44ab-8283-d4e750300326-plugins-config-volume\") pod \"8f8835f2-b94d-44ab-8283-d4e750300326\" (UID: \"8f8835f2-b94d-44ab-8283-d4e750300326\") "
Sep 10 23:20:54.600646 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.600588    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a-plugins-config-volume\") pod \"ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a\" (UID: \"ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a\") "
Sep 10 23:20:54.600711 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.600660    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvwkf\" (UniqueName: \"kubernetes.io/projected/8f8835f2-b94d-44ab-8283-d4e750300326-kube-api-access-dvwkf\") pod \"8f8835f2-b94d-44ab-8283-d4e750300326\" (UID: \"8f8835f2-b94d-44ab-8283-d4e750300326\") "
Sep 10 23:20:54.600915 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.600894    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8835f2-b94d-44ab-8283-d4e750300326-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f8835f2-b94d-44ab-8283-d4e750300326" (UID: "8f8835f2-b94d-44ab-8283-d4e750300326"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:20:54.601015 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.600992    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a" (UID: "ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:20:54.602577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.602548    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a-kube-api-access-mrr44" (OuterVolumeSpecName: "kube-api-access-mrr44") pod "ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a" (UID: "ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a"). InnerVolumeSpecName "kube-api-access-mrr44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:20:54.602658 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.602596    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8835f2-b94d-44ab-8283-d4e750300326-kube-api-access-dvwkf" (OuterVolumeSpecName: "kube-api-access-dvwkf") pod "8f8835f2-b94d-44ab-8283-d4e750300326" (UID: "8f8835f2-b94d-44ab-8283-d4e750300326"). InnerVolumeSpecName "kube-api-access-dvwkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:20:54.701451 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.701419    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:54.701451 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.701445    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvwkf\" (UniqueName: \"kubernetes.io/projected/8f8835f2-b94d-44ab-8283-d4e750300326-kube-api-access-dvwkf\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:54.701451 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.701456    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrr44\" (UniqueName: \"kubernetes.io/projected/ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a-kube-api-access-mrr44\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:54.701702 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.701464    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8835f2-b94d-44ab-8283-d4e750300326-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:20:54.807072 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.806983    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f8835f2-b94d-44ab-8283-d4e750300326" containerID="99deed1b1635aa5be34b5c5cd52b4f6cf9363bfa5613c900b5a4129796499ad2" exitCode=0
Sep 10 23:20:54.807072 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.807029    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"
Sep 10 23:20:54.807258 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.807067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw" event={"ID":"8f8835f2-b94d-44ab-8283-d4e750300326","Type":"ContainerDied","Data":"99deed1b1635aa5be34b5c5cd52b4f6cf9363bfa5613c900b5a4129796499ad2"}
Sep 10 23:20:54.807258 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.807120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw" event={"ID":"8f8835f2-b94d-44ab-8283-d4e750300326","Type":"ContainerDied","Data":"df88f352a6135218cc9adb0dc732ac5d270c72934690d1e81d94be223e42a721"}
Sep 10 23:20:54.807258 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.807141    2563 scope.go:117] "RemoveContainer" containerID="99deed1b1635aa5be34b5c5cd52b4f6cf9363bfa5613c900b5a4129796499ad2"
Sep 10 23:20:54.808712 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.808688    2563 generic.go:356] "Generic (PLEG): container finished" podID="ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a" containerID="f2d8719ab9e923b032a694f6709f87eae5b3b4d9edd51fe60d476089d052b9d6" exitCode=0
Sep 10 23:20:54.808836 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.808722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl" event={"ID":"ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a","Type":"ContainerDied","Data":"f2d8719ab9e923b032a694f6709f87eae5b3b4d9edd51fe60d476089d052b9d6"}
Sep 10 23:20:54.808836 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.808730    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"
Sep 10 23:20:54.808836 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.808759    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl" event={"ID":"ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a","Type":"ContainerDied","Data":"a15e86a063e79256180f20e9e2492a28c29bcc6921f42100e2256998f81970ae"}
Sep 10 23:20:54.817571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.817554    2563 scope.go:117] "RemoveContainer" containerID="99deed1b1635aa5be34b5c5cd52b4f6cf9363bfa5613c900b5a4129796499ad2"
Sep 10 23:20:54.817838 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:54.817819    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99deed1b1635aa5be34b5c5cd52b4f6cf9363bfa5613c900b5a4129796499ad2\": container with ID starting with 99deed1b1635aa5be34b5c5cd52b4f6cf9363bfa5613c900b5a4129796499ad2 not found: ID does not exist" containerID="99deed1b1635aa5be34b5c5cd52b4f6cf9363bfa5613c900b5a4129796499ad2"
Sep 10 23:20:54.817905 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.817845    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99deed1b1635aa5be34b5c5cd52b4f6cf9363bfa5613c900b5a4129796499ad2"} err="failed to get container status \"99deed1b1635aa5be34b5c5cd52b4f6cf9363bfa5613c900b5a4129796499ad2\": rpc error: code = NotFound desc = could not find container \"99deed1b1635aa5be34b5c5cd52b4f6cf9363bfa5613c900b5a4129796499ad2\": container with ID starting with 99deed1b1635aa5be34b5c5cd52b4f6cf9363bfa5613c900b5a4129796499ad2 not found: ID does not exist"
Sep 10 23:20:54.817905 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.817862    2563 scope.go:117] "RemoveContainer" containerID="f2d8719ab9e923b032a694f6709f87eae5b3b4d9edd51fe60d476089d052b9d6"
Sep 10 23:20:54.826007 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.825988    2563 scope.go:117] "RemoveContainer" containerID="f2d8719ab9e923b032a694f6709f87eae5b3b4d9edd51fe60d476089d052b9d6"
Sep 10 23:20:54.826264 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:20:54.826247    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d8719ab9e923b032a694f6709f87eae5b3b4d9edd51fe60d476089d052b9d6\": container with ID starting with f2d8719ab9e923b032a694f6709f87eae5b3b4d9edd51fe60d476089d052b9d6 not found: ID does not exist" containerID="f2d8719ab9e923b032a694f6709f87eae5b3b4d9edd51fe60d476089d052b9d6"
Sep 10 23:20:54.826315 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.826271    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d8719ab9e923b032a694f6709f87eae5b3b4d9edd51fe60d476089d052b9d6"} err="failed to get container status \"f2d8719ab9e923b032a694f6709f87eae5b3b4d9edd51fe60d476089d052b9d6\": rpc error: code = NotFound desc = could not find container \"f2d8719ab9e923b032a694f6709f87eae5b3b4d9edd51fe60d476089d052b9d6\": container with ID starting with f2d8719ab9e923b032a694f6709f87eae5b3b4d9edd51fe60d476089d052b9d6 not found: ID does not exist"
Sep 10 23:20:54.831750 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.831730    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"]
Sep 10 23:20:54.835050 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.835029    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-679c94-7f6f876966-pknkw"]
Sep 10 23:20:54.847894 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.847873    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"]
Sep 10 23:20:54.851448 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:54.851413    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-679c94-675dc79477-ckbhl"]
Sep 10 23:20:55.821759 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:55.821729    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f8835f2-b94d-44ab-8283-d4e750300326" path="/var/lib/kubelet/pods/8f8835f2-b94d-44ab-8283-d4e750300326/volumes"
Sep 10 23:20:55.822122 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:55.822046    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aebe214-3cbb-4de9-a43a-0ae9e1beba10" path="/var/lib/kubelet/pods/9aebe214-3cbb-4de9-a43a-0ae9e1beba10/volumes"
Sep 10 23:20:55.822366 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:55.822353    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a" path="/var/lib/kubelet/pods/ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a/volumes"
Sep 10 23:20:57.766633 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:20:57.766600    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"
Sep 10 23:21:19.854957 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:19.854928    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:21:19.856843 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:19.856820    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:21:19.939281 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:19.939255    2563 scope.go:117] "RemoveContainer" containerID="f0904e08fd606123ca44c7bd1cd382afc6dcc397dfc587f81127de82d82ae3da"
Sep 10 23:21:19.947628 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:19.947608    2563 scope.go:117] "RemoveContainer" containerID="23602df0388f16adfe5b025775b6c76109bb5c4f9c75a6a812d3abcedef2e9bf"
Sep 10 23:21:22.297357 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.297326    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d"]
Sep 10 23:21:22.297785 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.297771    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8835f2-b94d-44ab-8283-d4e750300326" containerName="payload-pre-processing"
Sep 10 23:21:22.297785 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.297786    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8835f2-b94d-44ab-8283-d4e750300326" containerName="payload-pre-processing"
Sep 10 23:21:22.297869 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.297796    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aebe214-3cbb-4de9-a43a-0ae9e1beba10" containerName="revoke-keys"
Sep 10 23:21:22.297869 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.297802    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aebe214-3cbb-4de9-a43a-0ae9e1beba10" containerName="revoke-keys"
Sep 10 23:21:22.297869 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.297819    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a" containerName="payload-processing"
Sep 10 23:21:22.297869 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.297825    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a" containerName="payload-processing"
Sep 10 23:21:22.297993 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.297890    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8835f2-b94d-44ab-8283-d4e750300326" containerName="payload-pre-processing"
Sep 10 23:21:22.297993 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.297897    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff15a1be-b7e3-4ceb-9c98-a5b75cfbbd0a" containerName="payload-processing"
Sep 10 23:21:22.297993 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.297903    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aebe214-3cbb-4de9-a43a-0ae9e1beba10" containerName="revoke-keys"
Sep 10 23:21:22.302578 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.302559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d"
Sep 10 23:21:22.305483 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.305456    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:21:22.305815 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.305792    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:21:22.308063 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.308043    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d"]
Sep 10 23:21:22.375401 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.375377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e868f72-ff69-4cdb-acf4-3365770c5f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d\" (UID: \"3e868f72-ff69-4cdb-acf4-3365770c5f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d"
Sep 10 23:21:22.476375 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.476343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e868f72-ff69-4cdb-acf4-3365770c5f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d\" (UID: \"3e868f72-ff69-4cdb-acf4-3365770c5f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d"
Sep 10 23:21:22.476960 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.476940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e868f72-ff69-4cdb-acf4-3365770c5f68-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d\" (UID: \"3e868f72-ff69-4cdb-acf4-3365770c5f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d"
Sep 10 23:21:22.614683 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.614653    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d"
Sep 10 23:21:22.883264 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.883229    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d"]
Sep 10 23:21:22.885412 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:21:22.885383    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e868f72_ff69_4cdb_acf4_3365770c5f68.slice/crio-ba886b686e961ab74e5658bcb8d0355ed7c83b2980f12f314cd51cf5608db9b9 WatchSource:0}: Error finding container ba886b686e961ab74e5658bcb8d0355ed7c83b2980f12f314cd51cf5608db9b9: Status 404 returned error can't find the container with id ba886b686e961ab74e5658bcb8d0355ed7c83b2980f12f314cd51cf5608db9b9
Sep 10 23:21:22.973528 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:22.973475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d" event={"ID":"3e868f72-ff69-4cdb-acf4-3365770c5f68","Type":"ContainerStarted","Data":"ba886b686e961ab74e5658bcb8d0355ed7c83b2980f12f314cd51cf5608db9b9"}
Sep 10 23:21:23.980804 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:23.980768    2563 generic.go:356] "Generic (PLEG): container finished" podID="3e868f72-ff69-4cdb-acf4-3365770c5f68" containerID="e7054d81da24f8ecec53862c9863ca9d47d372a756c7d11330ab917be4922dad" exitCode=0
Sep 10 23:21:23.981230 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:23.980848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d" event={"ID":"3e868f72-ff69-4cdb-acf4-3365770c5f68","Type":"ContainerDied","Data":"e7054d81da24f8ecec53862c9863ca9d47d372a756c7d11330ab917be4922dad"}
Sep 10 23:21:25.119644 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:25.119621    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d"
Sep 10 23:21:25.202320 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:25.202285    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e868f72-ff69-4cdb-acf4-3365770c5f68-maas-api-service-ca\") pod \"3e868f72-ff69-4cdb-acf4-3365770c5f68\" (UID: \"3e868f72-ff69-4cdb-acf4-3365770c5f68\") "
Sep 10 23:21:25.202646 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:25.202623    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e868f72-ff69-4cdb-acf4-3365770c5f68-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3e868f72-ff69-4cdb-acf4-3365770c5f68" (UID: "3e868f72-ff69-4cdb-acf4-3365770c5f68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:21:25.303431 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:25.303347    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e868f72-ff69-4cdb-acf4-3365770c5f68-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:25.996048 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:25.996016    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d"
Sep 10 23:21:25.996048 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:25.996046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d" event={"ID":"3e868f72-ff69-4cdb-acf4-3365770c5f68","Type":"ContainerDied","Data":"ba886b686e961ab74e5658bcb8d0355ed7c83b2980f12f314cd51cf5608db9b9"}
Sep 10 23:21:25.996299 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:25.996133    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba886b686e961ab74e5658bcb8d0355ed7c83b2980f12f314cd51cf5608db9b9"
Sep 10 23:21:27.169273 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:27.169236    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"]
Sep 10 23:21:27.173744 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:27.173717    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"]
Sep 10 23:21:32.924253 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:32.924209    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"]
Sep 10 23:21:32.929007 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:32.924474    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8" podUID="f0bb1063-d926-4635-8724-e585ecbc5285" containerName="payload-processing" containerID="cri-o://7cb91729343bfaa5e129b80f1ae6b315f1730149d405a3c03a1aca62c51aef27" gracePeriod=30
Sep 10 23:21:32.931893 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:32.931862    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"]
Sep 10 23:21:32.932367 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:32.932078    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx" podUID="018d3164-a9ab-42dc-b2a9-3ecb5349fe73" containerName="payload-pre-processing" containerID="cri-o://a3916a60a8cd0f441b9bee6059a2bbd1e124363eb33819f4e5930f22978921a1" gracePeriod=30
Sep 10 23:21:33.088669 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.088625    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d"]
Sep 10 23:21:33.092945 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.092917    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce17e-l5q9d"]
Sep 10 23:21:33.205586 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.205558    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"
Sep 10 23:21:33.208402 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.208378    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"
Sep 10 23:21:33.283895 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.283868    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv8ww\" (UniqueName: \"kubernetes.io/projected/f0bb1063-d926-4635-8724-e585ecbc5285-kube-api-access-rv8ww\") pod \"f0bb1063-d926-4635-8724-e585ecbc5285\" (UID: \"f0bb1063-d926-4635-8724-e585ecbc5285\") "
Sep 10 23:21:33.283895 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.283900    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bb1063-d926-4635-8724-e585ecbc5285-plugins-config-volume\") pod \"f0bb1063-d926-4635-8724-e585ecbc5285\" (UID: \"f0bb1063-d926-4635-8724-e585ecbc5285\") "
Sep 10 23:21:33.284100 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.283924    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/018d3164-a9ab-42dc-b2a9-3ecb5349fe73-plugins-config-volume\") pod \"018d3164-a9ab-42dc-b2a9-3ecb5349fe73\" (UID: \"018d3164-a9ab-42dc-b2a9-3ecb5349fe73\") "
Sep 10 23:21:33.284100 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.284016    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5smk\" (UniqueName: \"kubernetes.io/projected/018d3164-a9ab-42dc-b2a9-3ecb5349fe73-kube-api-access-r5smk\") pod \"018d3164-a9ab-42dc-b2a9-3ecb5349fe73\" (UID: \"018d3164-a9ab-42dc-b2a9-3ecb5349fe73\") "
Sep 10 23:21:33.284318 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.284289    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bb1063-d926-4635-8724-e585ecbc5285-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0bb1063-d926-4635-8724-e585ecbc5285" (UID: "f0bb1063-d926-4635-8724-e585ecbc5285"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:21:33.284393 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.284317    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018d3164-a9ab-42dc-b2a9-3ecb5349fe73-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "018d3164-a9ab-42dc-b2a9-3ecb5349fe73" (UID: "018d3164-a9ab-42dc-b2a9-3ecb5349fe73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:21:33.284442 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.284400    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0bb1063-d926-4635-8724-e585ecbc5285-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:33.284442 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.284413    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/018d3164-a9ab-42dc-b2a9-3ecb5349fe73-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:33.286134 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.286109    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bb1063-d926-4635-8724-e585ecbc5285-kube-api-access-rv8ww" (OuterVolumeSpecName: "kube-api-access-rv8ww") pod "f0bb1063-d926-4635-8724-e585ecbc5285" (UID: "f0bb1063-d926-4635-8724-e585ecbc5285"). InnerVolumeSpecName "kube-api-access-rv8ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:33.286233 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.286140    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018d3164-a9ab-42dc-b2a9-3ecb5349fe73-kube-api-access-r5smk" (OuterVolumeSpecName: "kube-api-access-r5smk") pod "018d3164-a9ab-42dc-b2a9-3ecb5349fe73" (UID: "018d3164-a9ab-42dc-b2a9-3ecb5349fe73"). InnerVolumeSpecName "kube-api-access-r5smk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:33.385827 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.385794    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5smk\" (UniqueName: \"kubernetes.io/projected/018d3164-a9ab-42dc-b2a9-3ecb5349fe73-kube-api-access-r5smk\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:33.385827 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.385821    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv8ww\" (UniqueName: \"kubernetes.io/projected/f0bb1063-d926-4635-8724-e585ecbc5285-kube-api-access-rv8ww\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:33.814001 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:33.813965    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e868f72-ff69-4cdb-acf4-3365770c5f68" path="/var/lib/kubelet/pods/3e868f72-ff69-4cdb-acf4-3365770c5f68/volumes"
Sep 10 23:21:34.042301 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.042270    2563 generic.go:356] "Generic (PLEG): container finished" podID="f0bb1063-d926-4635-8724-e585ecbc5285" containerID="7cb91729343bfaa5e129b80f1ae6b315f1730149d405a3c03a1aca62c51aef27" exitCode=0
Sep 10 23:21:34.042732 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.042321    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"
Sep 10 23:21:34.042732 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.042357    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8" event={"ID":"f0bb1063-d926-4635-8724-e585ecbc5285","Type":"ContainerDied","Data":"7cb91729343bfaa5e129b80f1ae6b315f1730149d405a3c03a1aca62c51aef27"}
Sep 10 23:21:34.042732 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.042410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8" event={"ID":"f0bb1063-d926-4635-8724-e585ecbc5285","Type":"ContainerDied","Data":"a6ff8b4baecb76f656e7ea6cafcd82685aef4eb969766321c48d718cba3c4722"}
Sep 10 23:21:34.042732 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.042438    2563 scope.go:117] "RemoveContainer" containerID="7cb91729343bfaa5e129b80f1ae6b315f1730149d405a3c03a1aca62c51aef27"
Sep 10 23:21:34.043936 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.043811    2563 generic.go:356] "Generic (PLEG): container finished" podID="018d3164-a9ab-42dc-b2a9-3ecb5349fe73" containerID="a3916a60a8cd0f441b9bee6059a2bbd1e124363eb33819f4e5930f22978921a1" exitCode=0
Sep 10 23:21:34.043936 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.043857    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"
Sep 10 23:21:34.043936 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.043887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx" event={"ID":"018d3164-a9ab-42dc-b2a9-3ecb5349fe73","Type":"ContainerDied","Data":"a3916a60a8cd0f441b9bee6059a2bbd1e124363eb33819f4e5930f22978921a1"}
Sep 10 23:21:34.043936 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.043927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx" event={"ID":"018d3164-a9ab-42dc-b2a9-3ecb5349fe73","Type":"ContainerDied","Data":"64895eb1318b91982924b6a92fd284051aa0338ad415fca7be38ddc1e49ae064"}
Sep 10 23:21:34.051626 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.051609    2563 scope.go:117] "RemoveContainer" containerID="7cb91729343bfaa5e129b80f1ae6b315f1730149d405a3c03a1aca62c51aef27"
Sep 10 23:21:34.051941 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:21:34.051919    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb91729343bfaa5e129b80f1ae6b315f1730149d405a3c03a1aca62c51aef27\": container with ID starting with 7cb91729343bfaa5e129b80f1ae6b315f1730149d405a3c03a1aca62c51aef27 not found: ID does not exist" containerID="7cb91729343bfaa5e129b80f1ae6b315f1730149d405a3c03a1aca62c51aef27"
Sep 10 23:21:34.051998 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.051950    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb91729343bfaa5e129b80f1ae6b315f1730149d405a3c03a1aca62c51aef27"} err="failed to get container status \"7cb91729343bfaa5e129b80f1ae6b315f1730149d405a3c03a1aca62c51aef27\": rpc error: code = NotFound desc = could not find container \"7cb91729343bfaa5e129b80f1ae6b315f1730149d405a3c03a1aca62c51aef27\": container with ID starting with 7cb91729343bfaa5e129b80f1ae6b315f1730149d405a3c03a1aca62c51aef27 not found: ID does not exist"
Sep 10 23:21:34.051998 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.051966    2563 scope.go:117] "RemoveContainer" containerID="a3916a60a8cd0f441b9bee6059a2bbd1e124363eb33819f4e5930f22978921a1"
Sep 10 23:21:34.060157 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.060137    2563 scope.go:117] "RemoveContainer" containerID="a3916a60a8cd0f441b9bee6059a2bbd1e124363eb33819f4e5930f22978921a1"
Sep 10 23:21:34.060416 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:21:34.060390    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3916a60a8cd0f441b9bee6059a2bbd1e124363eb33819f4e5930f22978921a1\": container with ID starting with a3916a60a8cd0f441b9bee6059a2bbd1e124363eb33819f4e5930f22978921a1 not found: ID does not exist" containerID="a3916a60a8cd0f441b9bee6059a2bbd1e124363eb33819f4e5930f22978921a1"
Sep 10 23:21:34.060487 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.060424    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3916a60a8cd0f441b9bee6059a2bbd1e124363eb33819f4e5930f22978921a1"} err="failed to get container status \"a3916a60a8cd0f441b9bee6059a2bbd1e124363eb33819f4e5930f22978921a1\": rpc error: code = NotFound desc = could not find container \"a3916a60a8cd0f441b9bee6059a2bbd1e124363eb33819f4e5930f22978921a1\": container with ID starting with a3916a60a8cd0f441b9bee6059a2bbd1e124363eb33819f4e5930f22978921a1 not found: ID does not exist"
Sep 10 23:21:34.061696 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.061677    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"]
Sep 10 23:21:34.068923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.068866    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-6ce17e-5b6fd8b698-9sxtx"]
Sep 10 23:21:34.078292 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.078270    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"]
Sep 10 23:21:34.082418 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:34.082397    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-6ce17e-696f5ff8d5-bdzk8"]
Sep 10 23:21:35.815241 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:35.815206    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="018d3164-a9ab-42dc-b2a9-3ecb5349fe73" path="/var/lib/kubelet/pods/018d3164-a9ab-42dc-b2a9-3ecb5349fe73/volumes"
Sep 10 23:21:35.815629 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:35.815615    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0bb1063-d926-4635-8724-e585ecbc5285" path="/var/lib/kubelet/pods/f0bb1063-d926-4635-8724-e585ecbc5285/volumes"
Sep 10 23:21:37.288968 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.288936    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955"]
Sep 10 23:21:37.289481 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.289462    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0bb1063-d926-4635-8724-e585ecbc5285" containerName="payload-processing"
Sep 10 23:21:37.289575 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.289485    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0bb1063-d926-4635-8724-e585ecbc5285" containerName="payload-processing"
Sep 10 23:21:37.289575 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.289510    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e868f72-ff69-4cdb-acf4-3365770c5f68" containerName="revoke-keys"
Sep 10 23:21:37.289575 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.289520    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e868f72-ff69-4cdb-acf4-3365770c5f68" containerName="revoke-keys"
Sep 10 23:21:37.289575 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.289531    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="018d3164-a9ab-42dc-b2a9-3ecb5349fe73" containerName="payload-pre-processing"
Sep 10 23:21:37.289575 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.289538    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="018d3164-a9ab-42dc-b2a9-3ecb5349fe73" containerName="payload-pre-processing"
Sep 10 23:21:37.289736 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.289607    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e868f72-ff69-4cdb-acf4-3365770c5f68" containerName="revoke-keys"
Sep 10 23:21:37.289736 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.289617    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0bb1063-d926-4635-8724-e585ecbc5285" containerName="payload-processing"
Sep 10 23:21:37.289736 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.289625    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="018d3164-a9ab-42dc-b2a9-3ecb5349fe73" containerName="payload-pre-processing"
Sep 10 23:21:37.294160 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.294143    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955"
Sep 10 23:21:37.297227 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.297200    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:21:37.297227 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.297204    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:21:37.300335 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.300308    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955"]
Sep 10 23:21:37.425738 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.425704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6680458a-8bce-4f44-96bd-958725d61b67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-568856-9v955\" (UID: \"6680458a-8bce-4f44-96bd-958725d61b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955"
Sep 10 23:21:37.526564 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.526530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6680458a-8bce-4f44-96bd-958725d61b67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-568856-9v955\" (UID: \"6680458a-8bce-4f44-96bd-958725d61b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955"
Sep 10 23:21:37.527157 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.527138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6680458a-8bce-4f44-96bd-958725d61b67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-568856-9v955\" (UID: \"6680458a-8bce-4f44-96bd-958725d61b67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955"
Sep 10 23:21:37.605358 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.605329    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955"
Sep 10 23:21:37.876342 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:37.876318    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955"]
Sep 10 23:21:37.878455 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:21:37.878428    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6680458a_8bce_4f44_96bd_958725d61b67.slice/crio-24e7214373dfd9a5a23439d9caeee0e8ab20d179d65b0321ef3d9ba744b98c76 WatchSource:0}: Error finding container 24e7214373dfd9a5a23439d9caeee0e8ab20d179d65b0321ef3d9ba744b98c76: Status 404 returned error can't find the container with id 24e7214373dfd9a5a23439d9caeee0e8ab20d179d65b0321ef3d9ba744b98c76
Sep 10 23:21:38.068911 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:38.068876    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955" event={"ID":"6680458a-8bce-4f44-96bd-958725d61b67","Type":"ContainerStarted","Data":"24e7214373dfd9a5a23439d9caeee0e8ab20d179d65b0321ef3d9ba744b98c76"}
Sep 10 23:21:39.076565 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:39.076532    2563 generic.go:356] "Generic (PLEG): container finished" podID="6680458a-8bce-4f44-96bd-958725d61b67" containerID="09d0c78e4122923de9db3bc49195250208ab26a0039e54aca095c6191f8ec1d4" exitCode=0
Sep 10 23:21:39.076949 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:39.076629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955" event={"ID":"6680458a-8bce-4f44-96bd-958725d61b67","Type":"ContainerDied","Data":"09d0c78e4122923de9db3bc49195250208ab26a0039e54aca095c6191f8ec1d4"}
Sep 10 23:21:40.216865 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:40.216840    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955"
Sep 10 23:21:40.355010 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:40.354924    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6680458a-8bce-4f44-96bd-958725d61b67-maas-api-service-ca\") pod \"6680458a-8bce-4f44-96bd-958725d61b67\" (UID: \"6680458a-8bce-4f44-96bd-958725d61b67\") "
Sep 10 23:21:40.355273 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:40.355250    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6680458a-8bce-4f44-96bd-958725d61b67-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6680458a-8bce-4f44-96bd-958725d61b67" (UID: "6680458a-8bce-4f44-96bd-958725d61b67"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:21:40.456044 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:40.456008    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6680458a-8bce-4f44-96bd-958725d61b67-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:41.088872 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:41.088832    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955" event={"ID":"6680458a-8bce-4f44-96bd-958725d61b67","Type":"ContainerDied","Data":"24e7214373dfd9a5a23439d9caeee0e8ab20d179d65b0321ef3d9ba744b98c76"}
Sep 10 23:21:41.089055 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:41.088885    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24e7214373dfd9a5a23439d9caeee0e8ab20d179d65b0321ef3d9ba744b98c76"
Sep 10 23:21:41.089055 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:41.088847    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955"
Sep 10 23:21:42.191442 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.191379    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff" podUID="ec235e1d-f185-45fa-9216-430298790d8c" containerName="main" containerID="cri-o://c96ddd403851911781891a9221ff86c8327c9291e17e20008fac9ce5c100cb67" gracePeriod=45
Sep 10 23:21:42.193137 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.193105    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92" podUID="32e515df-f4c6-4952-abd4-5421450cfdfd" containerName="main" containerID="cri-o://b21d05bd3357c6f95e2a30a11435f39d156df8e83043516c788b8f52e5b56db6" gracePeriod=45
Sep 10 23:21:42.477664 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.477635    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:21:42.480965 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.480944    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:21:42.577220 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577190    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-tmp-dir\") pod \"32e515df-f4c6-4952-abd4-5421450cfdfd\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") "
Sep 10 23:21:42.577390 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577231    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-tmp-dir\") pod \"ec235e1d-f185-45fa-9216-430298790d8c\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") "
Sep 10 23:21:42.577390 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577261    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-home\") pod \"32e515df-f4c6-4952-abd4-5421450cfdfd\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") "
Sep 10 23:21:42.577390 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577285    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-model-cache\") pod \"32e515df-f4c6-4952-abd4-5421450cfdfd\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") "
Sep 10 23:21:42.577390 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577301    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-model-cache\") pod \"ec235e1d-f185-45fa-9216-430298790d8c\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") "
Sep 10 23:21:42.577390 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577333    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfd7d\" (UniqueName: \"kubernetes.io/projected/32e515df-f4c6-4952-abd4-5421450cfdfd-kube-api-access-tfd7d\") pod \"32e515df-f4c6-4952-abd4-5421450cfdfd\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") "
Sep 10 23:21:42.577390 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577366    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec235e1d-f185-45fa-9216-430298790d8c-tls-certs\") pod \"ec235e1d-f185-45fa-9216-430298790d8c\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") "
Sep 10 23:21:42.577390 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577391    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e515df-f4c6-4952-abd4-5421450cfdfd-tls-certs\") pod \"32e515df-f4c6-4952-abd4-5421450cfdfd\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") "
Sep 10 23:21:42.577801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577418    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-home\") pod \"ec235e1d-f185-45fa-9216-430298790d8c\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") "
Sep 10 23:21:42.577801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577453    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-dshm\") pod \"ec235e1d-f185-45fa-9216-430298790d8c\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") "
Sep 10 23:21:42.577801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577473    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32e515df-f4c6-4952-abd4-5421450cfdfd" (UID: "32e515df-f4c6-4952-abd4-5421450cfdfd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.577801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577536    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-dshm\") pod \"32e515df-f4c6-4952-abd4-5421450cfdfd\" (UID: \"32e515df-f4c6-4952-abd4-5421450cfdfd\") "
Sep 10 23:21:42.577801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577553    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec235e1d-f185-45fa-9216-430298790d8c" (UID: "ec235e1d-f185-45fa-9216-430298790d8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.577801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577562    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-home" (OuterVolumeSpecName: "home") pod "32e515df-f4c6-4952-abd4-5421450cfdfd" (UID: "32e515df-f4c6-4952-abd4-5421450cfdfd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.577801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577573    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-model-cache" (OuterVolumeSpecName: "model-cache") pod "32e515df-f4c6-4952-abd4-5421450cfdfd" (UID: "32e515df-f4c6-4952-abd4-5421450cfdfd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.577801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577586    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6282m\" (UniqueName: \"kubernetes.io/projected/ec235e1d-f185-45fa-9216-430298790d8c-kube-api-access-6282m\") pod \"ec235e1d-f185-45fa-9216-430298790d8c\" (UID: \"ec235e1d-f185-45fa-9216-430298790d8c\") "
Sep 10 23:21:42.577801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577611    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec235e1d-f185-45fa-9216-430298790d8c" (UID: "ec235e1d-f185-45fa-9216-430298790d8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.578198 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.577814    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-home" (OuterVolumeSpecName: "home") pod "ec235e1d-f185-45fa-9216-430298790d8c" (UID: "ec235e1d-f185-45fa-9216-430298790d8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.578198 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.578001    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.578198 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.578022    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.578198 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.578038    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.578198 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.578051    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.578198 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.578065    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.578198 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.578077    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.579878 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.579854    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e515df-f4c6-4952-abd4-5421450cfdfd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32e515df-f4c6-4952-abd4-5421450cfdfd" (UID: "32e515df-f4c6-4952-abd4-5421450cfdfd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:21:42.579973 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.579902    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e515df-f4c6-4952-abd4-5421450cfdfd-kube-api-access-tfd7d" (OuterVolumeSpecName: "kube-api-access-tfd7d") pod "32e515df-f4c6-4952-abd4-5421450cfdfd" (UID: "32e515df-f4c6-4952-abd4-5421450cfdfd"). InnerVolumeSpecName "kube-api-access-tfd7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:42.580085 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.580064    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec235e1d-f185-45fa-9216-430298790d8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec235e1d-f185-45fa-9216-430298790d8c" (UID: "ec235e1d-f185-45fa-9216-430298790d8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:21:42.580132 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.580067    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-dshm" (OuterVolumeSpecName: "dshm") pod "32e515df-f4c6-4952-abd4-5421450cfdfd" (UID: "32e515df-f4c6-4952-abd4-5421450cfdfd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.580132 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.580091    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-dshm" (OuterVolumeSpecName: "dshm") pod "ec235e1d-f185-45fa-9216-430298790d8c" (UID: "ec235e1d-f185-45fa-9216-430298790d8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:21:42.580210 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.580132    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec235e1d-f185-45fa-9216-430298790d8c-kube-api-access-6282m" (OuterVolumeSpecName: "kube-api-access-6282m") pod "ec235e1d-f185-45fa-9216-430298790d8c" (UID: "ec235e1d-f185-45fa-9216-430298790d8c"). InnerVolumeSpecName "kube-api-access-6282m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:42.678847 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.678806    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfd7d\" (UniqueName: \"kubernetes.io/projected/32e515df-f4c6-4952-abd4-5421450cfdfd-kube-api-access-tfd7d\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.678847 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.678841    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec235e1d-f185-45fa-9216-430298790d8c-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.678847 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.678857    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e515df-f4c6-4952-abd4-5421450cfdfd-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.679083 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.678868    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec235e1d-f185-45fa-9216-430298790d8c-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.679083 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.678879    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e515df-f4c6-4952-abd4-5421450cfdfd-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:42.679083 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:42.678893    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6282m\" (UniqueName: \"kubernetes.io/projected/ec235e1d-f185-45fa-9216-430298790d8c-kube-api-access-6282m\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:43.100463 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.100427    2563 generic.go:356] "Generic (PLEG): container finished" podID="32e515df-f4c6-4952-abd4-5421450cfdfd" containerID="b21d05bd3357c6f95e2a30a11435f39d156df8e83043516c788b8f52e5b56db6" exitCode=0
Sep 10 23:21:43.100663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.100467    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"
Sep 10 23:21:43.100663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.100524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92" event={"ID":"32e515df-f4c6-4952-abd4-5421450cfdfd","Type":"ContainerDied","Data":"b21d05bd3357c6f95e2a30a11435f39d156df8e83043516c788b8f52e5b56db6"}
Sep 10 23:21:43.100663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.100566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92" event={"ID":"32e515df-f4c6-4952-abd4-5421450cfdfd","Type":"ContainerDied","Data":"526740c01e2df35231139fa1871dc28fd1da509a14cfbf7962c4fa24187ffd64"}
Sep 10 23:21:43.100663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.100590    2563 scope.go:117] "RemoveContainer" containerID="b21d05bd3357c6f95e2a30a11435f39d156df8e83043516c788b8f52e5b56db6"
Sep 10 23:21:43.102035 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.102012    2563 generic.go:356] "Generic (PLEG): container finished" podID="ec235e1d-f185-45fa-9216-430298790d8c" containerID="c96ddd403851911781891a9221ff86c8327c9291e17e20008fac9ce5c100cb67" exitCode=0
Sep 10 23:21:43.102147 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.102049    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"
Sep 10 23:21:43.102147 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.102102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff" event={"ID":"ec235e1d-f185-45fa-9216-430298790d8c","Type":"ContainerDied","Data":"c96ddd403851911781891a9221ff86c8327c9291e17e20008fac9ce5c100cb67"}
Sep 10 23:21:43.102147 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.102142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff" event={"ID":"ec235e1d-f185-45fa-9216-430298790d8c","Type":"ContainerDied","Data":"8956b0d9431930189f616ae2b075b00fed8f48e16033df2dc8c0ddf79a941c56"}
Sep 10 23:21:43.110432 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.110135    2563 scope.go:117] "RemoveContainer" containerID="b21d05bd3357c6f95e2a30a11435f39d156df8e83043516c788b8f52e5b56db6"
Sep 10 23:21:43.110698 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:21:43.110676    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21d05bd3357c6f95e2a30a11435f39d156df8e83043516c788b8f52e5b56db6\": container with ID starting with b21d05bd3357c6f95e2a30a11435f39d156df8e83043516c788b8f52e5b56db6 not found: ID does not exist" containerID="b21d05bd3357c6f95e2a30a11435f39d156df8e83043516c788b8f52e5b56db6"
Sep 10 23:21:43.110771 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.110705    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21d05bd3357c6f95e2a30a11435f39d156df8e83043516c788b8f52e5b56db6"} err="failed to get container status \"b21d05bd3357c6f95e2a30a11435f39d156df8e83043516c788b8f52e5b56db6\": rpc error: code = NotFound desc = could not find container \"b21d05bd3357c6f95e2a30a11435f39d156df8e83043516c788b8f52e5b56db6\": container with ID starting with b21d05bd3357c6f95e2a30a11435f39d156df8e83043516c788b8f52e5b56db6 not found: ID does not exist"
Sep 10 23:21:43.110771 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.110721    2563 scope.go:117] "RemoveContainer" containerID="c96ddd403851911781891a9221ff86c8327c9291e17e20008fac9ce5c100cb67"
Sep 10 23:21:43.118985 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.118971    2563 scope.go:117] "RemoveContainer" containerID="c96ddd403851911781891a9221ff86c8327c9291e17e20008fac9ce5c100cb67"
Sep 10 23:21:43.119229 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:21:43.119209    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96ddd403851911781891a9221ff86c8327c9291e17e20008fac9ce5c100cb67\": container with ID starting with c96ddd403851911781891a9221ff86c8327c9291e17e20008fac9ce5c100cb67 not found: ID does not exist" containerID="c96ddd403851911781891a9221ff86c8327c9291e17e20008fac9ce5c100cb67"
Sep 10 23:21:43.119298 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.119234    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96ddd403851911781891a9221ff86c8327c9291e17e20008fac9ce5c100cb67"} err="failed to get container status \"c96ddd403851911781891a9221ff86c8327c9291e17e20008fac9ce5c100cb67\": rpc error: code = NotFound desc = could not find container \"c96ddd403851911781891a9221ff86c8327c9291e17e20008fac9ce5c100cb67\": container with ID starting with c96ddd403851911781891a9221ff86c8327c9291e17e20008fac9ce5c100cb67 not found: ID does not exist"
Sep 10 23:21:43.124805 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.124783    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"]
Sep 10 23:21:43.126871 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.126849    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/premium-w0-6ce17e-kserve-6f49d6bbcf-2gl92"]
Sep 10 23:21:43.135018 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.134997    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"]
Sep 10 23:21:43.139172 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.139153    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6ce17e/distinct-w0-6ce17e-kserve-6788b999c4-t8xff"]
Sep 10 23:21:43.814540 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.814482    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32e515df-f4c6-4952-abd4-5421450cfdfd" path="/var/lib/kubelet/pods/32e515df-f4c6-4952-abd4-5421450cfdfd/volumes"
Sep 10 23:21:43.814934 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:43.814920    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec235e1d-f185-45fa-9216-430298790d8c" path="/var/lib/kubelet/pods/ec235e1d-f185-45fa-9216-430298790d8c/volumes"
Sep 10 23:21:47.807348 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:47.807305    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"]
Sep 10 23:21:47.807836 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:47.807610    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8" podUID="64ceb34f-b641-4ee5-a4a8-a8e27625f150" containerName="payload-processing" containerID="cri-o://ad09103f85270b8b48fda448f2a7724de42130a405477d406d120f3064b5f34d" gracePeriod=30
Sep 10 23:21:47.965050 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:47.965011    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955"]
Sep 10 23:21:47.967352 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:47.967324    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-568856-9v955"]
Sep 10 23:21:48.066802 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.066780    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"
Sep 10 23:21:48.133241 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.133201    2563 generic.go:356] "Generic (PLEG): container finished" podID="64ceb34f-b641-4ee5-a4a8-a8e27625f150" containerID="ad09103f85270b8b48fda448f2a7724de42130a405477d406d120f3064b5f34d" exitCode=0
Sep 10 23:21:48.133241 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.133242    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"
Sep 10 23:21:48.133452 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.133293    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8" event={"ID":"64ceb34f-b641-4ee5-a4a8-a8e27625f150","Type":"ContainerDied","Data":"ad09103f85270b8b48fda448f2a7724de42130a405477d406d120f3064b5f34d"}
Sep 10 23:21:48.133452 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.133343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8" event={"ID":"64ceb34f-b641-4ee5-a4a8-a8e27625f150","Type":"ContainerDied","Data":"2a66013f7ddf795414748bb3132c7d587319af675c855cccf5b44cdb66827ced"}
Sep 10 23:21:48.133452 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.133366    2563 scope.go:117] "RemoveContainer" containerID="ad09103f85270b8b48fda448f2a7724de42130a405477d406d120f3064b5f34d"
Sep 10 23:21:48.135610 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.135593    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf5pm\" (UniqueName: \"kubernetes.io/projected/64ceb34f-b641-4ee5-a4a8-a8e27625f150-kube-api-access-sf5pm\") pod \"64ceb34f-b641-4ee5-a4a8-a8e27625f150\" (UID: \"64ceb34f-b641-4ee5-a4a8-a8e27625f150\") "
Sep 10 23:21:48.135681 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.135623    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ceb34f-b641-4ee5-a4a8-a8e27625f150-plugins-config-volume\") pod \"64ceb34f-b641-4ee5-a4a8-a8e27625f150\" (UID: \"64ceb34f-b641-4ee5-a4a8-a8e27625f150\") "
Sep 10 23:21:48.136049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.136028    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ceb34f-b641-4ee5-a4a8-a8e27625f150-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64ceb34f-b641-4ee5-a4a8-a8e27625f150" (UID: "64ceb34f-b641-4ee5-a4a8-a8e27625f150"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:21:48.137662 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.137639    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ceb34f-b641-4ee5-a4a8-a8e27625f150-kube-api-access-sf5pm" (OuterVolumeSpecName: "kube-api-access-sf5pm") pod "64ceb34f-b641-4ee5-a4a8-a8e27625f150" (UID: "64ceb34f-b641-4ee5-a4a8-a8e27625f150"). InnerVolumeSpecName "kube-api-access-sf5pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:21:48.151431 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.151360    2563 scope.go:117] "RemoveContainer" containerID="ad09103f85270b8b48fda448f2a7724de42130a405477d406d120f3064b5f34d"
Sep 10 23:21:48.151991 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:21:48.151968    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad09103f85270b8b48fda448f2a7724de42130a405477d406d120f3064b5f34d\": container with ID starting with ad09103f85270b8b48fda448f2a7724de42130a405477d406d120f3064b5f34d not found: ID does not exist" containerID="ad09103f85270b8b48fda448f2a7724de42130a405477d406d120f3064b5f34d"
Sep 10 23:21:48.152075 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.152003    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad09103f85270b8b48fda448f2a7724de42130a405477d406d120f3064b5f34d"} err="failed to get container status \"ad09103f85270b8b48fda448f2a7724de42130a405477d406d120f3064b5f34d\": rpc error: code = NotFound desc = could not find container \"ad09103f85270b8b48fda448f2a7724de42130a405477d406d120f3064b5f34d\": container with ID starting with ad09103f85270b8b48fda448f2a7724de42130a405477d406d120f3064b5f34d not found: ID does not exist"
Sep 10 23:21:48.236807 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.236773    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf5pm\" (UniqueName: \"kubernetes.io/projected/64ceb34f-b641-4ee5-a4a8-a8e27625f150-kube-api-access-sf5pm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:48.236807 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.236805    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64ceb34f-b641-4ee5-a4a8-a8e27625f150-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:21:48.454388 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.454356    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"]
Sep 10 23:21:48.459707 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:48.459680    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-568856-8448cc7ff-cgvl8"]
Sep 10 23:21:49.814607 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:49.814574    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64ceb34f-b641-4ee5-a4a8-a8e27625f150" path="/var/lib/kubelet/pods/64ceb34f-b641-4ee5-a4a8-a8e27625f150/volumes"
Sep 10 23:21:49.814970 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:49.814903    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6680458a-8bce-4f44-96bd-958725d61b67" path="/var/lib/kubelet/pods/6680458a-8bce-4f44-96bd-958725d61b67/volumes"
Sep 10 23:21:51.333206 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.333163    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"]
Sep 10 23:21:51.333901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.333775    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ceb34f-b641-4ee5-a4a8-a8e27625f150" containerName="payload-processing"
Sep 10 23:21:51.333901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.333798    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ceb34f-b641-4ee5-a4a8-a8e27625f150" containerName="payload-processing"
Sep 10 23:21:51.333901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.333818    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6680458a-8bce-4f44-96bd-958725d61b67" containerName="revoke-keys"
Sep 10 23:21:51.333901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.333827    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6680458a-8bce-4f44-96bd-958725d61b67" containerName="revoke-keys"
Sep 10 23:21:51.333901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.333852    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32e515df-f4c6-4952-abd4-5421450cfdfd" containerName="main"
Sep 10 23:21:51.333901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.333861    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32e515df-f4c6-4952-abd4-5421450cfdfd" containerName="main"
Sep 10 23:21:51.333901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.333886    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec235e1d-f185-45fa-9216-430298790d8c" containerName="main"
Sep 10 23:21:51.333901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.333894    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec235e1d-f185-45fa-9216-430298790d8c" containerName="main"
Sep 10 23:21:51.334309 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.333981    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32e515df-f4c6-4952-abd4-5421450cfdfd" containerName="main"
Sep 10 23:21:51.334309 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.333993    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ceb34f-b641-4ee5-a4a8-a8e27625f150" containerName="payload-processing"
Sep 10 23:21:51.334309 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.334005    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec235e1d-f185-45fa-9216-430298790d8c" containerName="main"
Sep 10 23:21:51.334309 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.334019    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6680458a-8bce-4f44-96bd-958725d61b67" containerName="revoke-keys"
Sep 10 23:21:51.337034 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.337014    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.340231 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.340208    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c93e93\"/\"kube-root-ca.crt\""
Sep 10 23:21:51.340348 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.340212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c93e93\"/\"noten-model-c93e93-kserve-self-signed-certs\""
Sep 10 23:21:51.340348 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.340255    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c93e93\"/\"default-dockercfg-wtxpc\""
Sep 10 23:21:51.340348 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.340334    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c93e93\"/\"openshift-service-ca.crt\""
Sep 10 23:21:51.353554 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.348731    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"]
Sep 10 23:21:51.461291 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.461249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-model-cache\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.461291 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.461291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9206b8de-1bd0-41e2-8a39-185082cc985a-tls-certs\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.461562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.461403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sck4\" (UniqueName: \"kubernetes.io/projected/9206b8de-1bd0-41e2-8a39-185082cc985a-kube-api-access-6sck4\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.461562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.461446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-tmp-dir\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.461562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.461472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-dshm\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.461562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.461526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-home\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.562038 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.561976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sck4\" (UniqueName: \"kubernetes.io/projected/9206b8de-1bd0-41e2-8a39-185082cc985a-kube-api-access-6sck4\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.562198 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.562047    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-tmp-dir\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.562269 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.562207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-dshm\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.562335 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.562277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-home\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.562391 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.562347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-tmp-dir\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.562391 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.562358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-model-cache\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.562528 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.562460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9206b8de-1bd0-41e2-8a39-185082cc985a-tls-certs\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.562660 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.562639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-home\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.562730 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.562685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-model-cache\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.564376 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.564343    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-dshm\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.564681 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.564664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9206b8de-1bd0-41e2-8a39-185082cc985a-tls-certs\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.569566 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.569541    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sck4\" (UniqueName: \"kubernetes.io/projected/9206b8de-1bd0-41e2-8a39-185082cc985a-kube-api-access-6sck4\") pod \"noten-model-c93e93-kserve-84b84c56dc-49gxq\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") " pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.646639 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.646547    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:21:51.919205 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:51.919179    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"]
Sep 10 23:21:51.920827 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:21:51.920782    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9206b8de_1bd0_41e2_8a39_185082cc985a.slice/crio-976981bb4ca2110df95b0a1cda4c7a55edba0e72e4689013dcd013f24aee5fd7 WatchSource:0}: Error finding container 976981bb4ca2110df95b0a1cda4c7a55edba0e72e4689013dcd013f24aee5fd7: Status 404 returned error can't find the container with id 976981bb4ca2110df95b0a1cda4c7a55edba0e72e4689013dcd013f24aee5fd7
Sep 10 23:21:52.158230 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:52.158191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq" event={"ID":"9206b8de-1bd0-41e2-8a39-185082cc985a","Type":"ContainerStarted","Data":"a2efe551642d9553c34fed160a1141de207e45f0b32a5366b08ffab5272245d2"}
Sep 10 23:21:52.158230 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:52.158239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq" event={"ID":"9206b8de-1bd0-41e2-8a39-185082cc985a","Type":"ContainerStarted","Data":"976981bb4ca2110df95b0a1cda4c7a55edba0e72e4689013dcd013f24aee5fd7"}
Sep 10 23:21:52.176134 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:21:52.176030    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq" podStartSLOduration=1.176013002 podStartE2EDuration="1.176013002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:21:52.173703067 +0000 UTC m=+1232.865280251" watchObservedRunningTime="2026-09-10 23:21:52.176013002 +0000 UTC m=+1232.867590186"
Sep 10 23:22:01.646762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:01.646718    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:22:01.647161 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:01.646971    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:22:01.659440 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:01.659414    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:22:02.224127 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:02.224093    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:22:13.356175 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.356094    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"]
Sep 10 23:22:13.360006 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.359985    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.362318 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.362293    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"distinct-2-w4-7025b9-kserve-self-signed-certs\""
Sep 10 23:22:13.370175 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.370152    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"]
Sep 10 23:22:13.480471 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.480432    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-tls-certs\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.480471 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.480479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-tmp-dir\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.480719 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.480619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-dshm\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.480719 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.480676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvdhz\" (UniqueName: \"kubernetes.io/projected/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-kube-api-access-tvdhz\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.480719 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.480704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-home\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.480823 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.480741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-model-cache\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.581425 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.581373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-dshm\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.581629 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.581440    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvdhz\" (UniqueName: \"kubernetes.io/projected/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-kube-api-access-tvdhz\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.581629 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.581465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-home\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.581629 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.581526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-model-cache\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.581629 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.581582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-tls-certs\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.581629 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.581616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-tmp-dir\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.581958 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.581926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-home\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.581958 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.581943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-model-cache\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.582114 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.581986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-tmp-dir\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.583703 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.583681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-dshm\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.583982 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.583964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-tls-certs\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.589454 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.589432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvdhz\" (UniqueName: \"kubernetes.io/projected/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-kube-api-access-tvdhz\") pod \"distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") " pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.670427 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.670336    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:13.942065 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:13.942036    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"]
Sep 10 23:22:13.944752 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:22:13.944716    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dd338da_7bf3_45e5_85e1_94e9a6cbcd46.slice/crio-e119ac3966a65fa00564605f42d21936fa28465b3fded8d0615846a77a1bae9e WatchSource:0}: Error finding container e119ac3966a65fa00564605f42d21936fa28465b3fded8d0615846a77a1bae9e: Status 404 returned error can't find the container with id e119ac3966a65fa00564605f42d21936fa28465b3fded8d0615846a77a1bae9e
Sep 10 23:22:14.282739 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:14.282642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs" event={"ID":"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46","Type":"ContainerStarted","Data":"e189549095db1e4f82fcb13f4522730ca64db5f3310c0ded3f95cfb78655b344"}
Sep 10 23:22:14.282739 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:14.282689    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs" event={"ID":"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46","Type":"ContainerStarted","Data":"e119ac3966a65fa00564605f42d21936fa28465b3fded8d0615846a77a1bae9e"}
Sep 10 23:22:14.299754 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:14.299696    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs" podStartSLOduration=1.299679336 podStartE2EDuration="1.299679336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:22:14.297916086 +0000 UTC m=+1254.989493272" watchObservedRunningTime="2026-09-10 23:22:14.299679336 +0000 UTC m=+1254.991256519"
Sep 10 23:22:19.993995 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:19.993960    2563 scope.go:117] "RemoveContainer" containerID="434d5599a18f3a9a4a5efe9fca95dcf35f8f5369912096a83a677b5bc0e11b8b"
Sep 10 23:22:20.002801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:20.002780    2563 scope.go:117] "RemoveContainer" containerID="3ddf69411e8620110a0b8193401a5e2fdd14aac8ebcd205fccbdc112a2bd17f8"
Sep 10 23:22:23.670508 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:23.670472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:23.670986 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:23.670538    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:23.683454 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:23.683419    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:24.347283 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:24.347254    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:22:26.150012 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.149972    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"]
Sep 10 23:22:26.155776 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.155755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.158230 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.158205    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-76145153\"/\"kube-root-ca.crt\""
Sep 10 23:22:26.159363 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.159209    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-76145153\"/\"e2e-lifecycle-model-76145153-kserve-self-signed-certs\""
Sep 10 23:22:26.159363 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.159241    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-76145153\"/\"default-dockercfg-bpwmh\""
Sep 10 23:22:26.159363 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.159251    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-76145153\"/\"openshift-service-ca.crt\""
Sep 10 23:22:26.162566 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.162532    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"]
Sep 10 23:22:26.199681 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.199648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-tmp-dir\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.199848 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.199689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-home\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.199848 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.199715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-model-cache\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.199848 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.199788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-dshm\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.199848 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.199813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rf77\" (UniqueName: \"kubernetes.io/projected/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-kube-api-access-5rf77\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.199848 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.199846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-tls-certs\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.300462 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.300422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-home\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.300462 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.300471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-model-cache\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.300741 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.300535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-dshm\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.300741 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.300560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rf77\" (UniqueName: \"kubernetes.io/projected/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-kube-api-access-5rf77\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.300741 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.300614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-tls-certs\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.300741 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.300673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-tmp-dir\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.300951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.300840    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-home\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.300951 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.300892    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-model-cache\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.301049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.301001    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-tmp-dir\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.302857 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.302831    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-dshm\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.303136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.303118    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-tls-certs\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.308525 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.308485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rf77\" (UniqueName: \"kubernetes.io/projected/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-kube-api-access-5rf77\") pod \"e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") " pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.466785 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.466707    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:26.737549 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:26.737523    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"]
Sep 10 23:22:26.739162 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:22:26.739135    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0ef52a9_82b3_4d9f_9e5e_b23fa2a4321f.slice/crio-3b6bf0d97509d1254233e0747f0600a4c27f444ebe777a3d946d781a6faa034d WatchSource:0}: Error finding container 3b6bf0d97509d1254233e0747f0600a4c27f444ebe777a3d946d781a6faa034d: Status 404 returned error can't find the container with id 3b6bf0d97509d1254233e0747f0600a4c27f444ebe777a3d946d781a6faa034d
Sep 10 23:22:27.354168 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:27.354128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc" event={"ID":"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f","Type":"ContainerStarted","Data":"99554358383daf73cb6822c658042e9fddcdc3df923323f47922e4af089b432d"}
Sep 10 23:22:27.354662 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:27.354177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc" event={"ID":"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f","Type":"ContainerStarted","Data":"3b6bf0d97509d1254233e0747f0600a4c27f444ebe777a3d946d781a6faa034d"}
Sep 10 23:22:27.369652 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:27.369605    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc" podStartSLOduration=1.369590452 podStartE2EDuration="1.369590452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:22:27.368151241 +0000 UTC m=+1268.059728435" watchObservedRunningTime="2026-09-10 23:22:27.369590452 +0000 UTC m=+1268.061167636"
Sep 10 23:22:27.885136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:27.885092    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"]
Sep 10 23:22:36.223978 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.223943    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"]
Sep 10 23:22:36.227932 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.227910    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"
Sep 10 23:22:36.230169 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.230152    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-6c97f2\""
Sep 10 23:22:36.230239 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.230196    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-6c97f2-dockercfg-hxbs2\""
Sep 10 23:22:36.237990 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.237965    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"]
Sep 10 23:22:36.265018 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.264985    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"]
Sep 10 23:22:36.268464 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.268444    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"
Sep 10 23:22:36.276999 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.276974    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"]
Sep 10 23:22:36.301343 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.301309    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hlrg\" (UniqueName: \"kubernetes.io/projected/a47a9031-6ac1-4b11-b16c-15be5e73639f-kube-api-access-8hlrg\") pod \"payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn\" (UID: \"a47a9031-6ac1-4b11-b16c-15be5e73639f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"
Sep 10 23:22:36.301512 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.301347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47a9031-6ac1-4b11-b16c-15be5e73639f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn\" (UID: \"a47a9031-6ac1-4b11-b16c-15be5e73639f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"
Sep 10 23:22:36.402731 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.402700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5smc\" (UniqueName: \"kubernetes.io/projected/536c942a-7d7c-4a02-8f18-72279c461161-kube-api-access-b5smc\") pod \"payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm\" (UID: \"536c942a-7d7c-4a02-8f18-72279c461161\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"
Sep 10 23:22:36.402901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.402749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hlrg\" (UniqueName: \"kubernetes.io/projected/a47a9031-6ac1-4b11-b16c-15be5e73639f-kube-api-access-8hlrg\") pod \"payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn\" (UID: \"a47a9031-6ac1-4b11-b16c-15be5e73639f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"
Sep 10 23:22:36.402901 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.402775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47a9031-6ac1-4b11-b16c-15be5e73639f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn\" (UID: \"a47a9031-6ac1-4b11-b16c-15be5e73639f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"
Sep 10 23:22:36.402989 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.402910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536c942a-7d7c-4a02-8f18-72279c461161-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm\" (UID: \"536c942a-7d7c-4a02-8f18-72279c461161\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"
Sep 10 23:22:36.403383 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.403362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47a9031-6ac1-4b11-b16c-15be5e73639f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn\" (UID: \"a47a9031-6ac1-4b11-b16c-15be5e73639f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"
Sep 10 23:22:36.411320 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.411296    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hlrg\" (UniqueName: \"kubernetes.io/projected/a47a9031-6ac1-4b11-b16c-15be5e73639f-kube-api-access-8hlrg\") pod \"payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn\" (UID: \"a47a9031-6ac1-4b11-b16c-15be5e73639f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"
Sep 10 23:22:36.467393 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.467357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:36.467393 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.467403    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:36.480226 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.480148    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:36.503837 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.503808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536c942a-7d7c-4a02-8f18-72279c461161-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm\" (UID: \"536c942a-7d7c-4a02-8f18-72279c461161\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"
Sep 10 23:22:36.504022 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.503866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5smc\" (UniqueName: \"kubernetes.io/projected/536c942a-7d7c-4a02-8f18-72279c461161-kube-api-access-b5smc\") pod \"payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm\" (UID: \"536c942a-7d7c-4a02-8f18-72279c461161\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"
Sep 10 23:22:36.504596 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.504570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536c942a-7d7c-4a02-8f18-72279c461161-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm\" (UID: \"536c942a-7d7c-4a02-8f18-72279c461161\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"
Sep 10 23:22:36.511317 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.511295    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5smc\" (UniqueName: \"kubernetes.io/projected/536c942a-7d7c-4a02-8f18-72279c461161-kube-api-access-b5smc\") pod \"payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm\" (UID: \"536c942a-7d7c-4a02-8f18-72279c461161\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"
Sep 10 23:22:36.539204 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.539169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"
Sep 10 23:22:36.580694 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.580659    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"
Sep 10 23:22:36.828769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.828738    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"]
Sep 10 23:22:36.831381 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:22:36.831356    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47a9031_6ac1_4b11_b16c_15be5e73639f.slice/crio-eec8af806010cd155e25db0cbf55f260df95523eb2cfa1082551a2f0db36f033 WatchSource:0}: Error finding container eec8af806010cd155e25db0cbf55f260df95523eb2cfa1082551a2f0db36f033: Status 404 returned error can't find the container with id eec8af806010cd155e25db0cbf55f260df95523eb2cfa1082551a2f0db36f033
Sep 10 23:22:36.876059 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:36.876036    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"]
Sep 10 23:22:36.877743 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:22:36.877719    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536c942a_7d7c_4a02_8f18_72279c461161.slice/crio-e684500c39b14fcb8ae0fbe7e9a26a5cb06ed95b582b27d9311acad8063c688c WatchSource:0}: Error finding container e684500c39b14fcb8ae0fbe7e9a26a5cb06ed95b582b27d9311acad8063c688c: Status 404 returned error can't find the container with id e684500c39b14fcb8ae0fbe7e9a26a5cb06ed95b582b27d9311acad8063c688c
Sep 10 23:22:37.411642 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:37.411607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn" event={"ID":"a47a9031-6ac1-4b11-b16c-15be5e73639f","Type":"ContainerStarted","Data":"a41bae18700e9cb6fb7af8092ef069a44c429556ccd680ba09e7656d3b39ee72"}
Sep 10 23:22:37.412128 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:37.411671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn" event={"ID":"a47a9031-6ac1-4b11-b16c-15be5e73639f","Type":"ContainerStarted","Data":"eec8af806010cd155e25db0cbf55f260df95523eb2cfa1082551a2f0db36f033"}
Sep 10 23:22:37.412128 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:37.411700    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"
Sep 10 23:22:37.413424 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:37.413398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm" event={"ID":"536c942a-7d7c-4a02-8f18-72279c461161","Type":"ContainerStarted","Data":"c3a32612e964ce40cbc149443798ab10cd84abb8317a186b9f8dd13f7066a736"}
Sep 10 23:22:37.413543 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:37.413448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm" event={"ID":"536c942a-7d7c-4a02-8f18-72279c461161","Type":"ContainerStarted","Data":"e684500c39b14fcb8ae0fbe7e9a26a5cb06ed95b582b27d9311acad8063c688c"}
Sep 10 23:22:37.425220 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:37.425196    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:22:37.428478 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:37.428435    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn" podStartSLOduration=1.428421141 podStartE2EDuration="1.428421141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:22:37.426714896 +0000 UTC m=+1278.118292099" watchObservedRunningTime="2026-09-10 23:22:37.428421141 +0000 UTC m=+1278.119998324"
Sep 10 23:22:37.458455 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:37.458396    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm" podStartSLOduration=1.458375872 podStartE2EDuration="1.458375872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:22:37.457795182 +0000 UTC m=+1278.149372378" watchObservedRunningTime="2026-09-10 23:22:37.458375872 +0000 UTC m=+1278.149953058"
Sep 10 23:22:38.419486 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:38.419450    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"
Sep 10 23:22:42.904545 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:42.904503    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq" podUID="9206b8de-1bd0-41e2-8a39-185082cc985a" containerName="main" containerID="cri-o://a2efe551642d9553c34fed160a1141de207e45f0b32a5366b08ffab5272245d2" gracePeriod=45
Sep 10 23:22:43.162378 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.162312    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:22:43.274062 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.274027    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-dshm\") pod \"9206b8de-1bd0-41e2-8a39-185082cc985a\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") "
Sep 10 23:22:43.274246 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.274074    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9206b8de-1bd0-41e2-8a39-185082cc985a-tls-certs\") pod \"9206b8de-1bd0-41e2-8a39-185082cc985a\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") "
Sep 10 23:22:43.274246 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.274097    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-home\") pod \"9206b8de-1bd0-41e2-8a39-185082cc985a\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") "
Sep 10 23:22:43.274246 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.274199    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-tmp-dir\") pod \"9206b8de-1bd0-41e2-8a39-185082cc985a\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") "
Sep 10 23:22:43.274246 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.274229    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sck4\" (UniqueName: \"kubernetes.io/projected/9206b8de-1bd0-41e2-8a39-185082cc985a-kube-api-access-6sck4\") pod \"9206b8de-1bd0-41e2-8a39-185082cc985a\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") "
Sep 10 23:22:43.274467 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.274261    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-model-cache\") pod \"9206b8de-1bd0-41e2-8a39-185082cc985a\" (UID: \"9206b8de-1bd0-41e2-8a39-185082cc985a\") "
Sep 10 23:22:43.274467 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.274411    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-home" (OuterVolumeSpecName: "home") pod "9206b8de-1bd0-41e2-8a39-185082cc985a" (UID: "9206b8de-1bd0-41e2-8a39-185082cc985a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:22:43.275199 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.275127    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-model-cache" (OuterVolumeSpecName: "model-cache") pod "9206b8de-1bd0-41e2-8a39-185082cc985a" (UID: "9206b8de-1bd0-41e2-8a39-185082cc985a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:22:43.275199 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.275161    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9206b8de-1bd0-41e2-8a39-185082cc985a" (UID: "9206b8de-1bd0-41e2-8a39-185082cc985a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:22:43.275417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.275304    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:22:43.275417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.275326    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:22:43.275417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.275338    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:22:43.279832 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.277161    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9206b8de-1bd0-41e2-8a39-185082cc985a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9206b8de-1bd0-41e2-8a39-185082cc985a" (UID: "9206b8de-1bd0-41e2-8a39-185082cc985a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:22:43.279832 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.277631    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-dshm" (OuterVolumeSpecName: "dshm") pod "9206b8de-1bd0-41e2-8a39-185082cc985a" (UID: "9206b8de-1bd0-41e2-8a39-185082cc985a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:22:43.284100 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.284070    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9206b8de-1bd0-41e2-8a39-185082cc985a-kube-api-access-6sck4" (OuterVolumeSpecName: "kube-api-access-6sck4") pod "9206b8de-1bd0-41e2-8a39-185082cc985a" (UID: "9206b8de-1bd0-41e2-8a39-185082cc985a"). InnerVolumeSpecName "kube-api-access-6sck4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:22:43.376519 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.376458    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9206b8de-1bd0-41e2-8a39-185082cc985a-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:22:43.376519 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.376515    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9206b8de-1bd0-41e2-8a39-185082cc985a-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:22:43.376519 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.376528    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sck4\" (UniqueName: \"kubernetes.io/projected/9206b8de-1bd0-41e2-8a39-185082cc985a-kube-api-access-6sck4\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:22:43.448293 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.448202    2563 generic.go:356] "Generic (PLEG): container finished" podID="9206b8de-1bd0-41e2-8a39-185082cc985a" containerID="a2efe551642d9553c34fed160a1141de207e45f0b32a5366b08ffab5272245d2" exitCode=0
Sep 10 23:22:43.448293 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.448261    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"
Sep 10 23:22:43.448484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.448293    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq" event={"ID":"9206b8de-1bd0-41e2-8a39-185082cc985a","Type":"ContainerDied","Data":"a2efe551642d9553c34fed160a1141de207e45f0b32a5366b08ffab5272245d2"}
Sep 10 23:22:43.448484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.448340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq" event={"ID":"9206b8de-1bd0-41e2-8a39-185082cc985a","Type":"ContainerDied","Data":"976981bb4ca2110df95b0a1cda4c7a55edba0e72e4689013dcd013f24aee5fd7"}
Sep 10 23:22:43.448484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.448358    2563 scope.go:117] "RemoveContainer" containerID="a2efe551642d9553c34fed160a1141de207e45f0b32a5366b08ffab5272245d2"
Sep 10 23:22:43.457831 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.457811    2563 scope.go:117] "RemoveContainer" containerID="a2efe551642d9553c34fed160a1141de207e45f0b32a5366b08ffab5272245d2"
Sep 10 23:22:43.458120 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:22:43.458098    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2efe551642d9553c34fed160a1141de207e45f0b32a5366b08ffab5272245d2\": container with ID starting with a2efe551642d9553c34fed160a1141de207e45f0b32a5366b08ffab5272245d2 not found: ID does not exist" containerID="a2efe551642d9553c34fed160a1141de207e45f0b32a5366b08ffab5272245d2"
Sep 10 23:22:43.458190 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.458128    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2efe551642d9553c34fed160a1141de207e45f0b32a5366b08ffab5272245d2"} err="failed to get container status \"a2efe551642d9553c34fed160a1141de207e45f0b32a5366b08ffab5272245d2\": rpc error: code = NotFound desc = could not find container \"a2efe551642d9553c34fed160a1141de207e45f0b32a5366b08ffab5272245d2\": container with ID starting with a2efe551642d9553c34fed160a1141de207e45f0b32a5366b08ffab5272245d2 not found: ID does not exist"
Sep 10 23:22:43.468610 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.468581    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"]
Sep 10 23:22:43.472242 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.472221    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c93e93/noten-model-c93e93-kserve-84b84c56dc-49gxq"]
Sep 10 23:22:43.815295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:43.815266    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9206b8de-1bd0-41e2-8a39-185082cc985a" path="/var/lib/kubelet/pods/9206b8de-1bd0-41e2-8a39-185082cc985a/volumes"
Sep 10 23:22:48.422077 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:48.422046    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"
Sep 10 23:22:49.426613 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:22:49.426580    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"
Sep 10 23:23:04.298865 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.298830    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"]
Sep 10 23:23:04.299305 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.299292    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9206b8de-1bd0-41e2-8a39-185082cc985a" containerName="main"
Sep 10 23:23:04.299305 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.299307    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9206b8de-1bd0-41e2-8a39-185082cc985a" containerName="main"
Sep 10 23:23:04.299402 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.299393    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9206b8de-1bd0-41e2-8a39-185082cc985a" containerName="main"
Sep 10 23:23:04.302722 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.302703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"
Sep 10 23:23:04.305012 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.304991    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-83be66\""
Sep 10 23:23:04.313483 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.313461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"]
Sep 10 23:23:04.345340 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.345306    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"]
Sep 10 23:23:04.348932 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.348910    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"
Sep 10 23:23:04.351153 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.351128    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-83be66-dockercfg-68xnb\""
Sep 10 23:23:04.356308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.356177    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"]
Sep 10 23:23:04.378262 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.378232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pncgm\" (UniqueName: \"kubernetes.io/projected/67636cdc-1284-4dc0-801c-4d90ee4e5c67-kube-api-access-pncgm\") pod \"payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2\" (UID: \"67636cdc-1284-4dc0-801c-4d90ee4e5c67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"
Sep 10 23:23:04.378412 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.378296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c238769d-e4f5-443a-aa41-26cfb97da683-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr\" (UID: \"c238769d-e4f5-443a-aa41-26cfb97da683\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"
Sep 10 23:23:04.378412 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.378317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67636cdc-1284-4dc0-801c-4d90ee4e5c67-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2\" (UID: \"67636cdc-1284-4dc0-801c-4d90ee4e5c67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"
Sep 10 23:23:04.378412 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.378374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdnvd\" (UniqueName: \"kubernetes.io/projected/c238769d-e4f5-443a-aa41-26cfb97da683-kube-api-access-xdnvd\") pod \"payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr\" (UID: \"c238769d-e4f5-443a-aa41-26cfb97da683\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"
Sep 10 23:23:04.479938 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.479890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdnvd\" (UniqueName: \"kubernetes.io/projected/c238769d-e4f5-443a-aa41-26cfb97da683-kube-api-access-xdnvd\") pod \"payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr\" (UID: \"c238769d-e4f5-443a-aa41-26cfb97da683\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"
Sep 10 23:23:04.480143 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.479975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pncgm\" (UniqueName: \"kubernetes.io/projected/67636cdc-1284-4dc0-801c-4d90ee4e5c67-kube-api-access-pncgm\") pod \"payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2\" (UID: \"67636cdc-1284-4dc0-801c-4d90ee4e5c67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"
Sep 10 23:23:04.480143 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.480041    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c238769d-e4f5-443a-aa41-26cfb97da683-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr\" (UID: \"c238769d-e4f5-443a-aa41-26cfb97da683\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"
Sep 10 23:23:04.480143 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.480066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67636cdc-1284-4dc0-801c-4d90ee4e5c67-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2\" (UID: \"67636cdc-1284-4dc0-801c-4d90ee4e5c67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"
Sep 10 23:23:04.480842 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.480817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67636cdc-1284-4dc0-801c-4d90ee4e5c67-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2\" (UID: \"67636cdc-1284-4dc0-801c-4d90ee4e5c67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"
Sep 10 23:23:04.480944 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.480825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c238769d-e4f5-443a-aa41-26cfb97da683-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr\" (UID: \"c238769d-e4f5-443a-aa41-26cfb97da683\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"
Sep 10 23:23:04.487671 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.487651    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pncgm\" (UniqueName: \"kubernetes.io/projected/67636cdc-1284-4dc0-801c-4d90ee4e5c67-kube-api-access-pncgm\") pod \"payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2\" (UID: \"67636cdc-1284-4dc0-801c-4d90ee4e5c67\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"
Sep 10 23:23:04.487769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.487748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdnvd\" (UniqueName: \"kubernetes.io/projected/c238769d-e4f5-443a-aa41-26cfb97da683-kube-api-access-xdnvd\") pod \"payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr\" (UID: \"c238769d-e4f5-443a-aa41-26cfb97da683\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"
Sep 10 23:23:04.614615 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.614578    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"
Sep 10 23:23:04.660482 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.660450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"
Sep 10 23:23:04.885644 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.885617    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"]
Sep 10 23:23:04.887783 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:23:04.887754    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67636cdc_1284_4dc0_801c_4d90ee4e5c67.slice/crio-47f1fe3a4cd19674555607586899594d97131442f571059c74034f7746dc2af7 WatchSource:0}: Error finding container 47f1fe3a4cd19674555607586899594d97131442f571059c74034f7746dc2af7: Status 404 returned error can't find the container with id 47f1fe3a4cd19674555607586899594d97131442f571059c74034f7746dc2af7
Sep 10 23:23:04.951340 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:04.951315    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"]
Sep 10 23:23:04.952799 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:23:04.952774    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc238769d_e4f5_443a_aa41_26cfb97da683.slice/crio-f9aa34489aeb065abd7072e4f4a1480570ae7a489250ba418ff3120b951a0a38 WatchSource:0}: Error finding container f9aa34489aeb065abd7072e4f4a1480570ae7a489250ba418ff3120b951a0a38: Status 404 returned error can't find the container with id f9aa34489aeb065abd7072e4f4a1480570ae7a489250ba418ff3120b951a0a38
Sep 10 23:23:05.571654 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:05.571620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2" event={"ID":"67636cdc-1284-4dc0-801c-4d90ee4e5c67","Type":"ContainerStarted","Data":"160427bbf34802a716999af583b63387da491735ab65b50fae21b9963afe4b25"}
Sep 10 23:23:05.572125 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:05.571683    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"
Sep 10 23:23:05.572125 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:05.571705    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2" event={"ID":"67636cdc-1284-4dc0-801c-4d90ee4e5c67","Type":"ContainerStarted","Data":"47f1fe3a4cd19674555607586899594d97131442f571059c74034f7746dc2af7"}
Sep 10 23:23:05.573306 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:05.573277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr" event={"ID":"c238769d-e4f5-443a-aa41-26cfb97da683","Type":"ContainerStarted","Data":"f8544377577d97683b60b4ca6fff3fe6625bec9e309f1dc9be6a8ac2137ea2ce"}
Sep 10 23:23:05.573440 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:05.573330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr" event={"ID":"c238769d-e4f5-443a-aa41-26cfb97da683","Type":"ContainerStarted","Data":"f9aa34489aeb065abd7072e4f4a1480570ae7a489250ba418ff3120b951a0a38"}
Sep 10 23:23:05.573440 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:05.573360    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"
Sep 10 23:23:05.587773 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:05.587720    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2" podStartSLOduration=1.587705326 podStartE2EDuration="1.587705326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:23:05.586908282 +0000 UTC m=+1306.278485488" watchObservedRunningTime="2026-09-10 23:23:05.587705326 +0000 UTC m=+1306.279282510"
Sep 10 23:23:05.602308 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:05.602269    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr" podStartSLOduration=1.602255726 podStartE2EDuration="1.602255726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:23:05.602037339 +0000 UTC m=+1306.293614524" watchObservedRunningTime="2026-09-10 23:23:05.602255726 +0000 UTC m=+1306.293832909"
Sep 10 23:23:16.581120 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:16.581084    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"
Sep 10 23:23:16.581601 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:16.581265    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"
Sep 10 23:23:20.046512 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:20.046459    2563 scope.go:117] "RemoveContainer" containerID="6bfa3585d5d4290eb9045f5e9d2c1ecf7b4e72febf25a79186270bfa03fe991d"
Sep 10 23:23:20.055400 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:20.055376    2563 scope.go:117] "RemoveContainer" containerID="5e6f7d4ed8c5de2159433d34384f5a6549a3a686f9eee2397dddeb9d76fa3a09"
Sep 10 23:23:36.206152 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:36.206112    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn"]
Sep 10 23:23:36.211523 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:36.211479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn"
Sep 10 23:23:36.214242 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:36.214187    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:23:36.214242 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:36.214208    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:23:36.216749 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:36.216722    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn"]
Sep 10 23:23:36.299348 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:36.299315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a4f101-39af-4774-b437-d4f1bc9abce0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-76145153-stpkn\" (UID: \"30a4f101-39af-4774-b437-d4f1bc9abce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn"
Sep 10 23:23:36.400801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:36.400760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a4f101-39af-4774-b437-d4f1bc9abce0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-76145153-stpkn\" (UID: \"30a4f101-39af-4774-b437-d4f1bc9abce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn"
Sep 10 23:23:36.401439 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:36.401412    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a4f101-39af-4774-b437-d4f1bc9abce0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-76145153-stpkn\" (UID: \"30a4f101-39af-4774-b437-d4f1bc9abce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn"
Sep 10 23:23:36.525235 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:36.525123    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn"
Sep 10 23:23:36.796540 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:36.796515    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn"]
Sep 10 23:23:36.799608 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:23:36.799565    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a4f101_39af_4774_b437_d4f1bc9abce0.slice/crio-7247c7db50176174c1fb7230c054dfb01249858d2ea23a4934dc783a9b2cd877 WatchSource:0}: Error finding container 7247c7db50176174c1fb7230c054dfb01249858d2ea23a4934dc783a9b2cd877: Status 404 returned error can't find the container with id 7247c7db50176174c1fb7230c054dfb01249858d2ea23a4934dc783a9b2cd877
Sep 10 23:23:37.752124 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:37.752093    2563 generic.go:356] "Generic (PLEG): container finished" podID="30a4f101-39af-4774-b437-d4f1bc9abce0" containerID="972b2c66c6945d2b705f054ed4185402432da9a1be155a5f09255a39a2bf9ba7" exitCode=0
Sep 10 23:23:37.752484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:37.752172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn" event={"ID":"30a4f101-39af-4774-b437-d4f1bc9abce0","Type":"ContainerDied","Data":"972b2c66c6945d2b705f054ed4185402432da9a1be155a5f09255a39a2bf9ba7"}
Sep 10 23:23:37.752484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:37.752237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn" event={"ID":"30a4f101-39af-4774-b437-d4f1bc9abce0","Type":"ContainerStarted","Data":"7247c7db50176174c1fb7230c054dfb01249858d2ea23a4934dc783a9b2cd877"}
Sep 10 23:23:38.888269 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:38.888246    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn"
Sep 10 23:23:38.924475 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:38.924444    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a4f101-39af-4774-b437-d4f1bc9abce0-maas-api-service-ca\") pod \"30a4f101-39af-4774-b437-d4f1bc9abce0\" (UID: \"30a4f101-39af-4774-b437-d4f1bc9abce0\") "
Sep 10 23:23:38.924852 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:38.924833    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a4f101-39af-4774-b437-d4f1bc9abce0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30a4f101-39af-4774-b437-d4f1bc9abce0" (UID: "30a4f101-39af-4774-b437-d4f1bc9abce0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:23:39.025907 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:39.025822    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30a4f101-39af-4774-b437-d4f1bc9abce0-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:23:39.764242 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:39.764214    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn"
Sep 10 23:23:39.764417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:39.764262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn" event={"ID":"30a4f101-39af-4774-b437-d4f1bc9abce0","Type":"ContainerDied","Data":"7247c7db50176174c1fb7230c054dfb01249858d2ea23a4934dc783a9b2cd877"}
Sep 10 23:23:39.764417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:39.764316    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7247c7db50176174c1fb7230c054dfb01249858d2ea23a4934dc783a9b2cd877"
Sep 10 23:23:51.762099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:51.761308    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"]
Sep 10 23:23:51.762545 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:51.762267    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd" podUID="39bffac5-509f-4094-9091-4b48041bd4f7" containerName="payload-pre-processing" containerID="cri-o://f723f7ae3f19eb3febc5d02bd6551c901891564303f75aa86caa52998c324df4" gracePeriod=30
Sep 10 23:23:51.906441 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:51.906400    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn"]
Sep 10 23:23:51.908716 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:51.908687    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76145153-stpkn"]
Sep 10 23:23:52.013252 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.013193    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"
Sep 10 23:23:52.158724 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.158684    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlq2q\" (UniqueName: \"kubernetes.io/projected/39bffac5-509f-4094-9091-4b48041bd4f7-kube-api-access-vlq2q\") pod \"39bffac5-509f-4094-9091-4b48041bd4f7\" (UID: \"39bffac5-509f-4094-9091-4b48041bd4f7\") "
Sep 10 23:23:52.158930 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.158850    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39bffac5-509f-4094-9091-4b48041bd4f7-plugins-config-volume\") pod \"39bffac5-509f-4094-9091-4b48041bd4f7\" (UID: \"39bffac5-509f-4094-9091-4b48041bd4f7\") "
Sep 10 23:23:52.159212 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.159185    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39bffac5-509f-4094-9091-4b48041bd4f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39bffac5-509f-4094-9091-4b48041bd4f7" (UID: "39bffac5-509f-4094-9091-4b48041bd4f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:23:52.161027 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.161002    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bffac5-509f-4094-9091-4b48041bd4f7-kube-api-access-vlq2q" (OuterVolumeSpecName: "kube-api-access-vlq2q") pod "39bffac5-509f-4094-9091-4b48041bd4f7" (UID: "39bffac5-509f-4094-9091-4b48041bd4f7"). InnerVolumeSpecName "kube-api-access-vlq2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:23:52.259668 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.259630    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39bffac5-509f-4094-9091-4b48041bd4f7-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:23:52.259668 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.259666    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlq2q\" (UniqueName: \"kubernetes.io/projected/39bffac5-509f-4094-9091-4b48041bd4f7-kube-api-access-vlq2q\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:23:52.839296 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.839264    2563 generic.go:356] "Generic (PLEG): container finished" podID="39bffac5-509f-4094-9091-4b48041bd4f7" containerID="f723f7ae3f19eb3febc5d02bd6551c901891564303f75aa86caa52998c324df4" exitCode=0
Sep 10 23:23:52.839720 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.839305    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"
Sep 10 23:23:52.839720 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.839356    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd" event={"ID":"39bffac5-509f-4094-9091-4b48041bd4f7","Type":"ContainerDied","Data":"f723f7ae3f19eb3febc5d02bd6551c901891564303f75aa86caa52998c324df4"}
Sep 10 23:23:52.839720 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.839404    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd" event={"ID":"39bffac5-509f-4094-9091-4b48041bd4f7","Type":"ContainerDied","Data":"0b39b157f85b3e492afe4f57ef17bba2fcdc86f57091766a6820d618303a335e"}
Sep 10 23:23:52.839720 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.839427    2563 scope.go:117] "RemoveContainer" containerID="f723f7ae3f19eb3febc5d02bd6551c901891564303f75aa86caa52998c324df4"
Sep 10 23:23:52.848836 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.848811    2563 scope.go:117] "RemoveContainer" containerID="f723f7ae3f19eb3febc5d02bd6551c901891564303f75aa86caa52998c324df4"
Sep 10 23:23:52.849130 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:23:52.849111    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f723f7ae3f19eb3febc5d02bd6551c901891564303f75aa86caa52998c324df4\": container with ID starting with f723f7ae3f19eb3febc5d02bd6551c901891564303f75aa86caa52998c324df4 not found: ID does not exist" containerID="f723f7ae3f19eb3febc5d02bd6551c901891564303f75aa86caa52998c324df4"
Sep 10 23:23:52.849190 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.849139    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f723f7ae3f19eb3febc5d02bd6551c901891564303f75aa86caa52998c324df4"} err="failed to get container status \"f723f7ae3f19eb3febc5d02bd6551c901891564303f75aa86caa52998c324df4\": rpc error: code = NotFound desc = could not find container \"f723f7ae3f19eb3febc5d02bd6551c901891564303f75aa86caa52998c324df4\": container with ID starting with f723f7ae3f19eb3febc5d02bd6551c901891564303f75aa86caa52998c324df4 not found: ID does not exist"
Sep 10 23:23:52.862275 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.862253    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"]
Sep 10 23:23:52.867982 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:52.867957    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-76145153-5d9f749df8-8xmbd"]
Sep 10 23:23:53.815385 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:53.815344    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30a4f101-39af-4774-b437-d4f1bc9abce0" path="/var/lib/kubelet/pods/30a4f101-39af-4774-b437-d4f1bc9abce0/volumes"
Sep 10 23:23:53.815987 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:53.815966    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39bffac5-509f-4094-9091-4b48041bd4f7" path="/var/lib/kubelet/pods/39bffac5-509f-4094-9091-4b48041bd4f7/volumes"
Sep 10 23:23:59.919283 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:23:59.919248    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"]
Sep 10 23:24:09.960448 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:09.960405    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"]
Sep 10 23:24:09.960979 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:09.960854    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39bffac5-509f-4094-9091-4b48041bd4f7" containerName="payload-pre-processing"
Sep 10 23:24:09.960979 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:09.960868    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="39bffac5-509f-4094-9091-4b48041bd4f7" containerName="payload-pre-processing"
Sep 10 23:24:09.960979 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:09.960876    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a4f101-39af-4774-b437-d4f1bc9abce0" containerName="revoke-keys"
Sep 10 23:24:09.960979 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:09.960882    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a4f101-39af-4774-b437-d4f1bc9abce0" containerName="revoke-keys"
Sep 10 23:24:09.960979 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:09.960953    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="39bffac5-509f-4094-9091-4b48041bd4f7" containerName="payload-pre-processing"
Sep 10 23:24:09.960979 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:09.960964    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a4f101-39af-4774-b437-d4f1bc9abce0" containerName="revoke-keys"
Sep 10 23:24:09.965247 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:09.965230    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:09.968424 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:09.968405    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6c97f2\"/\"default-dockercfg-vxzdz\""
Sep 10 23:24:09.968424 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:09.968411    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6c97f2\"/\"auth-test-model-6c97f2-kserve-self-signed-certs\""
Sep 10 23:24:09.968632 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:09.968412    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6c97f2\"/\"kube-root-ca.crt\""
Sep 10 23:24:09.968632 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:09.968462    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6c97f2\"/\"openshift-service-ca.crt\""
Sep 10 23:24:09.972218 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:09.972191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"]
Sep 10 23:24:10.035844 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.035806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-tls-certs\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.036037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.035874    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-home\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.036037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.035904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-tmp-dir\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.036037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.035933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-dshm\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.036037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.036023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-model-cache\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.036270 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.036131    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk7nh\" (UniqueName: \"kubernetes.io/projected/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-kube-api-access-gk7nh\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.136701 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.136663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-model-cache\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.136889 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.136763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk7nh\" (UniqueName: \"kubernetes.io/projected/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-kube-api-access-gk7nh\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.136889 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.136819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-tls-certs\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.136889 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.136845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-home\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.136889 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.136870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-tmp-dir\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.137076 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.136898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-dshm\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.137136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.137066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-model-cache\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.137237 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.137211    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-home\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.137301 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.137237    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-tmp-dir\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.139115 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.139093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-dshm\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.139295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.139277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-tls-certs\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.144959 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.144931    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk7nh\" (UniqueName: \"kubernetes.io/projected/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-kube-api-access-gk7nh\") pod \"auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.275749 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.275647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:10.548443 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.548419    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"]
Sep 10 23:24:10.550051 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:24:10.550017    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05ddd8f1_de97_4ad9_be78_e68f04c0e88b.slice/crio-90109ddaa3518c230fb07e7ef289a6376f98cd9ca1d1c63bce21af649dbc989b WatchSource:0}: Error finding container 90109ddaa3518c230fb07e7ef289a6376f98cd9ca1d1c63bce21af649dbc989b: Status 404 returned error can't find the container with id 90109ddaa3518c230fb07e7ef289a6376f98cd9ca1d1c63bce21af649dbc989b
Sep 10 23:24:10.943104 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.943071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk" event={"ID":"05ddd8f1-de97-4ad9-be78-e68f04c0e88b","Type":"ContainerStarted","Data":"69c8023f15be0552ff5fef7a89ce62ccf99defe717b03422dd3f02fc89982962"}
Sep 10 23:24:10.943104 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.943111    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk" event={"ID":"05ddd8f1-de97-4ad9-be78-e68f04c0e88b","Type":"ContainerStarted","Data":"90109ddaa3518c230fb07e7ef289a6376f98cd9ca1d1c63bce21af649dbc989b"}
Sep 10 23:24:10.958163 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:10.958110    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk" podStartSLOduration=1.958095645 podStartE2EDuration="1.958095645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:24:10.957586864 +0000 UTC m=+1371.649164060" watchObservedRunningTime="2026-09-10 23:24:10.958095645 +0000 UTC m=+1371.649672829"
Sep 10 23:24:14.938618 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:14.938556    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc" podUID="c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f" containerName="main" containerID="cri-o://99554358383daf73cb6822c658042e9fddcdc3df923323f47922e4af089b432d" gracePeriod=45
Sep 10 23:24:15.176949 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.176922    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:24:15.285426 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.285348    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rf77\" (UniqueName: \"kubernetes.io/projected/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-kube-api-access-5rf77\") pod \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") "
Sep 10 23:24:15.285426 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.285420    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-dshm\") pod \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") "
Sep 10 23:24:15.285637 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.285447    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-home\") pod \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") "
Sep 10 23:24:15.285637 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.285471    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-tls-certs\") pod \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") "
Sep 10 23:24:15.285637 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.285532    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-model-cache\") pod \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") "
Sep 10 23:24:15.285637 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.285573    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-tmp-dir\") pod \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\" (UID: \"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f\") "
Sep 10 23:24:15.285820 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.285738    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-home" (OuterVolumeSpecName: "home") pod "c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f" (UID: "c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:24:15.285886 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.285827    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f" (UID: "c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:24:15.285943 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.285901    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f" (UID: "c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:24:15.285982 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.285934    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:24:15.285982 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.285961    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:24:15.287649 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.287626    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f" (UID: "c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:24:15.287757 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.287658    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-dshm" (OuterVolumeSpecName: "dshm") pod "c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f" (UID: "c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:24:15.287757 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.287712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-kube-api-access-5rf77" (OuterVolumeSpecName: "kube-api-access-5rf77") pod "c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f" (UID: "c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f"). InnerVolumeSpecName "kube-api-access-5rf77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:24:15.386955 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.386918    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:24:15.386955 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.386951    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rf77\" (UniqueName: \"kubernetes.io/projected/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-kube-api-access-5rf77\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:24:15.386955 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.386959    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:24:15.387191 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.386968    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:24:15.978834 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.978799    2563 generic.go:356] "Generic (PLEG): container finished" podID="c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f" containerID="99554358383daf73cb6822c658042e9fddcdc3df923323f47922e4af089b432d" exitCode=0
Sep 10 23:24:15.979252 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.978841    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"
Sep 10 23:24:15.979252 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.978887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc" event={"ID":"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f","Type":"ContainerDied","Data":"99554358383daf73cb6822c658042e9fddcdc3df923323f47922e4af089b432d"}
Sep 10 23:24:15.979252 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.978930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc" event={"ID":"c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f","Type":"ContainerDied","Data":"3b6bf0d97509d1254233e0747f0600a4c27f444ebe777a3d946d781a6faa034d"}
Sep 10 23:24:15.979252 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.978950    2563 scope.go:117] "RemoveContainer" containerID="99554358383daf73cb6822c658042e9fddcdc3df923323f47922e4af089b432d"
Sep 10 23:24:15.987830 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.987812    2563 scope.go:117] "RemoveContainer" containerID="99554358383daf73cb6822c658042e9fddcdc3df923323f47922e4af089b432d"
Sep 10 23:24:15.988082 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:24:15.988064    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99554358383daf73cb6822c658042e9fddcdc3df923323f47922e4af089b432d\": container with ID starting with 99554358383daf73cb6822c658042e9fddcdc3df923323f47922e4af089b432d not found: ID does not exist" containerID="99554358383daf73cb6822c658042e9fddcdc3df923323f47922e4af089b432d"
Sep 10 23:24:15.988150 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.988095    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99554358383daf73cb6822c658042e9fddcdc3df923323f47922e4af089b432d"} err="failed to get container status \"99554358383daf73cb6822c658042e9fddcdc3df923323f47922e4af089b432d\": rpc error: code = NotFound desc = could not find container \"99554358383daf73cb6822c658042e9fddcdc3df923323f47922e4af089b432d\": container with ID starting with 99554358383daf73cb6822c658042e9fddcdc3df923323f47922e4af089b432d not found: ID does not exist"
Sep 10 23:24:15.993284 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.993260    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"]
Sep 10 23:24:15.997136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:15.997110    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-76145153/e2e-lifecycle-model-76145153-kserve-64ff746c47-nxktc"]
Sep 10 23:24:17.815404 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:17.815366    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f" path="/var/lib/kubelet/pods/c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f/volumes"
Sep 10 23:24:20.276237 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:20.276199    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:20.276746 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:20.276261    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:20.288558 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:20.288528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:21.019796 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:21.019765    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:24:46.560115 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.560077    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"]
Sep 10 23:24:46.560733 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.560558    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f" containerName="main"
Sep 10 23:24:46.560733 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.560572    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f" containerName="main"
Sep 10 23:24:46.560733 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.560651    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0ef52a9-82b3-4d9f-9e5e-b23fa2a4321f" containerName="main"
Sep 10 23:24:46.565010 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.564991    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.567225 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.567202    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"embedding-w4-7025b9-kserve-self-signed-certs\""
Sep 10 23:24:46.574161 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.574139    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"]
Sep 10 23:24:46.611579 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.611547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-model-cache\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.611705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.611608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-home\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.611705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.611640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/548169f7-e180-41e8-bd45-2755dc4ed316-tls-certs\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.611705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.611655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pffkc\" (UniqueName: \"kubernetes.io/projected/548169f7-e180-41e8-bd45-2755dc4ed316-kube-api-access-pffkc\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.611831 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.611732    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-tmp-dir\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.611831 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.611756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-dshm\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.713034 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.712998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-model-cache\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.713206 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.713066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-home\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.713206 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.713100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/548169f7-e180-41e8-bd45-2755dc4ed316-tls-certs\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.713206 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.713122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pffkc\" (UniqueName: \"kubernetes.io/projected/548169f7-e180-41e8-bd45-2755dc4ed316-kube-api-access-pffkc\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.713206 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.713171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-tmp-dir\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.713206 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.713196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-dshm\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.713477 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.713428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-model-cache\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.713582 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.713478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-home\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.713582 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.713529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-tmp-dir\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.715398 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.715363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-dshm\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.715663 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.715645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/548169f7-e180-41e8-bd45-2755dc4ed316-tls-certs\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.720579 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.720557    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pffkc\" (UniqueName: \"kubernetes.io/projected/548169f7-e180-41e8-bd45-2755dc4ed316-kube-api-access-pffkc\") pod \"embedding-w4-7025b9-kserve-7544fb77c-f24g2\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") " pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:46.874230 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:46.874139    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:47.147885 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:47.147859    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"]
Sep 10 23:24:47.148984 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:24:47.148954    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod548169f7_e180_41e8_bd45_2755dc4ed316.slice/crio-f685d624b840013e9f5fcdb0e8222c56c676d6e9beb3d4bc18eb575d0851731f WatchSource:0}: Error finding container f685d624b840013e9f5fcdb0e8222c56c676d6e9beb3d4bc18eb575d0851731f: Status 404 returned error can't find the container with id f685d624b840013e9f5fcdb0e8222c56c676d6e9beb3d4bc18eb575d0851731f
Sep 10 23:24:48.160749 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:48.160706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2" event={"ID":"548169f7-e180-41e8-bd45-2755dc4ed316","Type":"ContainerStarted","Data":"1bbd27cb96c7e4ae0036ded507cf6d0b0422f9e9aef36c73cfda3f4558adf718"}
Sep 10 23:24:48.160749 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:48.160756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2" event={"ID":"548169f7-e180-41e8-bd45-2755dc4ed316","Type":"ContainerStarted","Data":"f685d624b840013e9f5fcdb0e8222c56c676d6e9beb3d4bc18eb575d0851731f"}
Sep 10 23:24:48.178449 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:48.178404    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2" podStartSLOduration=2.178389405 podStartE2EDuration="2.178389405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:24:48.176167037 +0000 UTC m=+1408.867744222" watchObservedRunningTime="2026-09-10 23:24:48.178389405 +0000 UTC m=+1408.869966636"
Sep 10 23:24:56.875203 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:56.875164    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:56.875203 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:56.875211    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:56.887488 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:56.887463    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:24:57.224125 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:24:57.224050    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:25:20.103161 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:20.103091    2563 scope.go:117] "RemoveContainer" containerID="9c1bb5978b6a5674f157ee27b38549b381dca02673ab4a688207232a5c50e4c3"
Sep 10 23:25:57.469836 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:57.469801    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"]
Sep 10 23:25:57.480298 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:57.480270    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"
Sep 10 23:25:57.482600 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:57.482566    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a4c77c\""
Sep 10 23:25:57.483231 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:57.482614    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a4c77c-dockercfg-75xkk\""
Sep 10 23:25:57.485110 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:57.485089    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"]
Sep 10 23:25:57.604424 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:57.604391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee993953-b6e4-456f-8a8a-fec7734e29fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj\" (UID: \"ee993953-b6e4-456f-8a8a-fec7734e29fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"
Sep 10 23:25:57.604597 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:57.604435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr8z7\" (UniqueName: \"kubernetes.io/projected/ee993953-b6e4-456f-8a8a-fec7734e29fa-kube-api-access-sr8z7\") pod \"payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj\" (UID: \"ee993953-b6e4-456f-8a8a-fec7734e29fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"
Sep 10 23:25:57.705252 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:57.705217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee993953-b6e4-456f-8a8a-fec7734e29fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj\" (UID: \"ee993953-b6e4-456f-8a8a-fec7734e29fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"
Sep 10 23:25:57.705410 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:57.705262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr8z7\" (UniqueName: \"kubernetes.io/projected/ee993953-b6e4-456f-8a8a-fec7734e29fa-kube-api-access-sr8z7\") pod \"payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj\" (UID: \"ee993953-b6e4-456f-8a8a-fec7734e29fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"
Sep 10 23:25:57.706029 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:57.706010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee993953-b6e4-456f-8a8a-fec7734e29fa-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj\" (UID: \"ee993953-b6e4-456f-8a8a-fec7734e29fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"
Sep 10 23:25:57.713561 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:57.713534    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr8z7\" (UniqueName: \"kubernetes.io/projected/ee993953-b6e4-456f-8a8a-fec7734e29fa-kube-api-access-sr8z7\") pod \"payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj\" (UID: \"ee993953-b6e4-456f-8a8a-fec7734e29fa\") " pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"
Sep 10 23:25:57.793642 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:57.793571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"
Sep 10 23:25:58.065657 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:58.065631    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"]
Sep 10 23:25:58.067370 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:25:58.067344    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee993953_b6e4_456f_8a8a_fec7734e29fa.slice/crio-b67e563c29c33b6ae8bbfa0f9a3bb3248dd3ace66f83d8dae76f7f14dcf97ebf WatchSource:0}: Error finding container b67e563c29c33b6ae8bbfa0f9a3bb3248dd3ace66f83d8dae76f7f14dcf97ebf: Status 404 returned error can't find the container with id b67e563c29c33b6ae8bbfa0f9a3bb3248dd3ace66f83d8dae76f7f14dcf97ebf
Sep 10 23:25:58.546527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:58.546405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj" event={"ID":"ee993953-b6e4-456f-8a8a-fec7734e29fa","Type":"ContainerStarted","Data":"14638fc6477970a5c976b3112326470ced09f7d0302f1d593d7b484d5d9aa9a7"}
Sep 10 23:25:58.546527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:58.546459    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj" event={"ID":"ee993953-b6e4-456f-8a8a-fec7734e29fa","Type":"ContainerStarted","Data":"b67e563c29c33b6ae8bbfa0f9a3bb3248dd3ace66f83d8dae76f7f14dcf97ebf"}
Sep 10 23:25:58.547018 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:58.546552    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"
Sep 10 23:25:58.564292 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:25:58.564237    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj" podStartSLOduration=1.564218643 podStartE2EDuration="1.564218643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:25:58.562020686 +0000 UTC m=+1479.253597895" watchObservedRunningTime="2026-09-10 23:25:58.564218643 +0000 UTC m=+1479.255795831"
Sep 10 23:26:09.555667 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:09.555632    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"
Sep 10 23:26:13.636878 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.636840    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"]
Sep 10 23:26:13.640887 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.640863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.643168 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.643147    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7025b9\"/\"e2e-conflict-b-d74b488b-kserve-self-signed-certs\""
Sep 10 23:26:13.647654 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.647628    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"]
Sep 10 23:26:13.754556 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.754517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/318faa10-7505-494c-8081-7ef4c53b9a6f-tls-certs\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.754762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.754625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-dshm\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.754762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.754700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-home\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.754762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.754749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcpvx\" (UniqueName: \"kubernetes.io/projected/318faa10-7505-494c-8081-7ef4c53b9a6f-kube-api-access-fcpvx\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.754930 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.754806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-tmp-dir\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.754930 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.754849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-model-cache\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.855853 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.855822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/318faa10-7505-494c-8081-7ef4c53b9a6f-tls-certs\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.856017 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.855883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-dshm\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.856017 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.855935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-home\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.856017 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.855969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpvx\" (UniqueName: \"kubernetes.io/projected/318faa10-7505-494c-8081-7ef4c53b9a6f-kube-api-access-fcpvx\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.856017 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.856008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-tmp-dir\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.856256 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.856048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-model-cache\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.856396 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.856370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-home\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.856474 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.856414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-model-cache\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.856561 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.856532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-tmp-dir\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.858155 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.858132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-dshm\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.858417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.858402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/318faa10-7505-494c-8081-7ef4c53b9a6f-tls-certs\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.863361 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.863340    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcpvx\" (UniqueName: \"kubernetes.io/projected/318faa10-7505-494c-8081-7ef4c53b9a6f-kube-api-access-fcpvx\") pod \"e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") " pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:13.950774 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:13.950701    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:14.220265 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:14.220239    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"]
Sep 10 23:26:14.222751 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:26:14.222720    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod318faa10_7505_494c_8081_7ef4c53b9a6f.slice/crio-c8ca4807cda038c3419eaa9acf7bda9a8985c6d7bbbaca16e88b73fef2f4b593 WatchSource:0}: Error finding container c8ca4807cda038c3419eaa9acf7bda9a8985c6d7bbbaca16e88b73fef2f4b593: Status 404 returned error can't find the container with id c8ca4807cda038c3419eaa9acf7bda9a8985c6d7bbbaca16e88b73fef2f4b593
Sep 10 23:26:14.636126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:14.636084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz" event={"ID":"318faa10-7505-494c-8081-7ef4c53b9a6f","Type":"ContainerStarted","Data":"097d754a98973ff9ad607db65d0e81cdcc1a7a3b25bb27d0c6ac7a0ef8f9e088"}
Sep 10 23:26:14.636126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:14.636139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz" event={"ID":"318faa10-7505-494c-8081-7ef4c53b9a6f","Type":"ContainerStarted","Data":"c8ca4807cda038c3419eaa9acf7bda9a8985c6d7bbbaca16e88b73fef2f4b593"}
Sep 10 23:26:14.653508 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:14.653448    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz" podStartSLOduration=1.653433122 podStartE2EDuration="1.653433122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:26:14.650751096 +0000 UTC m=+1495.342328353" watchObservedRunningTime="2026-09-10 23:26:14.653433122 +0000 UTC m=+1495.345010358"
Sep 10 23:26:19.891642 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:19.891617    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:26:19.893435 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:19.893410    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:26:23.951561 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:23.951531    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:23.951944 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:23.951605    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:23.964266 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:23.964243    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:24.704152 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:24.704125    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:26.621689 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:26.621656    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"]
Sep 10 23:26:26.625580 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:26.625557    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"
Sep 10 23:26:26.628092 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:26.628066    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-72f076-dockercfg-rrqtx\""
Sep 10 23:26:26.628209 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:26.628074    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-72f076\""
Sep 10 23:26:26.635618 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:26.635591    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"]
Sep 10 23:26:26.691091 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:26.691048    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10900502-db8a-4c8e-9354-0cb50b786f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm\" (UID: \"10900502-db8a-4c8e-9354-0cb50b786f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"
Sep 10 23:26:26.691258 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:26.691202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-852sz\" (UniqueName: \"kubernetes.io/projected/10900502-db8a-4c8e-9354-0cb50b786f0e-kube-api-access-852sz\") pod \"payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm\" (UID: \"10900502-db8a-4c8e-9354-0cb50b786f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"
Sep 10 23:26:26.792374 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:26.792338    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10900502-db8a-4c8e-9354-0cb50b786f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm\" (UID: \"10900502-db8a-4c8e-9354-0cb50b786f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"
Sep 10 23:26:26.792546 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:26.792482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852sz\" (UniqueName: \"kubernetes.io/projected/10900502-db8a-4c8e-9354-0cb50b786f0e-kube-api-access-852sz\") pod \"payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm\" (UID: \"10900502-db8a-4c8e-9354-0cb50b786f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"
Sep 10 23:26:26.793132 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:26.793113    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10900502-db8a-4c8e-9354-0cb50b786f0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm\" (UID: \"10900502-db8a-4c8e-9354-0cb50b786f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"
Sep 10 23:26:26.804277 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:26.804248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-852sz\" (UniqueName: \"kubernetes.io/projected/10900502-db8a-4c8e-9354-0cb50b786f0e-kube-api-access-852sz\") pod \"payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm\" (UID: \"10900502-db8a-4c8e-9354-0cb50b786f0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"
Sep 10 23:26:26.938257 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:26.938182    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"
Sep 10 23:26:27.209948 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:27.209912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"]
Sep 10 23:26:27.213514 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:26:27.213466    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10900502_db8a_4c8e_9354_0cb50b786f0e.slice/crio-86f3bb79bf63c482e22e2bd106fd23fd672d94934a85db68733682f872613798 WatchSource:0}: Error finding container 86f3bb79bf63c482e22e2bd106fd23fd672d94934a85db68733682f872613798: Status 404 returned error can't find the container with id 86f3bb79bf63c482e22e2bd106fd23fd672d94934a85db68733682f872613798
Sep 10 23:26:27.710748 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:27.710707    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm" event={"ID":"10900502-db8a-4c8e-9354-0cb50b786f0e","Type":"ContainerStarted","Data":"a62196f828f07a91bd26ac147db86ad900c98508a10b63141d5e162c35ba8dfe"}
Sep 10 23:26:27.711148 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:27.710770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm" event={"ID":"10900502-db8a-4c8e-9354-0cb50b786f0e","Type":"ContainerStarted","Data":"86f3bb79bf63c482e22e2bd106fd23fd672d94934a85db68733682f872613798"}
Sep 10 23:26:27.711148 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:27.710945    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"
Sep 10 23:26:27.729310 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:27.729261    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm" podStartSLOduration=1.7292476730000002 podStartE2EDuration="1.729247673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:26:27.727003186 +0000 UTC m=+1508.418580392" watchObservedRunningTime="2026-09-10 23:26:27.729247673 +0000 UTC m=+1508.420824908"
Sep 10 23:26:38.718367 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:38.718333    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"
Sep 10 23:26:43.233054 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:43.232978    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"]
Sep 10 23:26:52.543168 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:52.543124    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5"]
Sep 10 23:26:52.546893 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:52.546877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5"
Sep 10 23:26:52.549409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:52.549390    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:26:52.549510 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:52.549396    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:26:52.555085 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:52.555062    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5"]
Sep 10 23:26:52.652567 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:52.652535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5\" (UID: \"01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5"
Sep 10 23:26:52.753986 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:52.753943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5\" (UID: \"01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5"
Sep 10 23:26:52.754616 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:52.754597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5\" (UID: \"01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5"
Sep 10 23:26:52.858338 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:52.858312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5"
Sep 10 23:26:53.130592 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:53.130566    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5"]
Sep 10 23:26:53.132684 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:26:53.132657    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d6157d_8f5c_4c6a_837b_1f3a0fd83b0f.slice/crio-ce397e53029408efb44c4699ac178388bc65abc70ffc90b4b228839ed81e92c3 WatchSource:0}: Error finding container ce397e53029408efb44c4699ac178388bc65abc70ffc90b4b228839ed81e92c3: Status 404 returned error can't find the container with id ce397e53029408efb44c4699ac178388bc65abc70ffc90b4b228839ed81e92c3
Sep 10 23:26:53.855354 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:53.855326    2563 generic.go:356] "Generic (PLEG): container finished" podID="01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f" containerID="644c0248f7c57018d9528b17acb1a40378ad9aa8684a1d1b0f61919b5253934f" exitCode=0
Sep 10 23:26:53.855772 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:53.855414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5" event={"ID":"01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f","Type":"ContainerDied","Data":"644c0248f7c57018d9528b17acb1a40378ad9aa8684a1d1b0f61919b5253934f"}
Sep 10 23:26:53.855772 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:53.855458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5" event={"ID":"01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f","Type":"ContainerStarted","Data":"ce397e53029408efb44c4699ac178388bc65abc70ffc90b4b228839ed81e92c3"}
Sep 10 23:26:54.997532 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:54.997509    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5"
Sep 10 23:26:55.077589 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:55.077558    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f-maas-api-service-ca\") pod \"01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f\" (UID: \"01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f\") "
Sep 10 23:26:55.077929 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:55.077906    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f" (UID: "01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:26:55.178750 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:55.178671    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:26:55.867707 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:55.867670    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5"
Sep 10 23:26:55.867856 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:55.867697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5" event={"ID":"01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f","Type":"ContainerDied","Data":"ce397e53029408efb44c4699ac178388bc65abc70ffc90b4b228839ed81e92c3"}
Sep 10 23:26:55.867856 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:55.867753    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce397e53029408efb44c4699ac178388bc65abc70ffc90b4b228839ed81e92c3"
Sep 10 23:26:58.252729 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.252688    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz" podUID="318faa10-7505-494c-8081-7ef4c53b9a6f" containerName="main" containerID="cri-o://097d754a98973ff9ad607db65d0e81cdcc1a7a3b25bb27d0c6ac7a0ef8f9e088" gracePeriod=45
Sep 10 23:26:58.505816 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.505751    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:58.615927 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.615890    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-tmp-dir\") pod \"318faa10-7505-494c-8081-7ef4c53b9a6f\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") "
Sep 10 23:26:58.615927 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.615931    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-model-cache\") pod \"318faa10-7505-494c-8081-7ef4c53b9a6f\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") "
Sep 10 23:26:58.616150 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.615963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-dshm\") pod \"318faa10-7505-494c-8081-7ef4c53b9a6f\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") "
Sep 10 23:26:58.616150 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.616011    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcpvx\" (UniqueName: \"kubernetes.io/projected/318faa10-7505-494c-8081-7ef4c53b9a6f-kube-api-access-fcpvx\") pod \"318faa10-7505-494c-8081-7ef4c53b9a6f\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") "
Sep 10 23:26:58.616150 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.616041    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/318faa10-7505-494c-8081-7ef4c53b9a6f-tls-certs\") pod \"318faa10-7505-494c-8081-7ef4c53b9a6f\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") "
Sep 10 23:26:58.616150 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.616076    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-home\") pod \"318faa10-7505-494c-8081-7ef4c53b9a6f\" (UID: \"318faa10-7505-494c-8081-7ef4c53b9a6f\") "
Sep 10 23:26:58.616372 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.616256    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-model-cache" (OuterVolumeSpecName: "model-cache") pod "318faa10-7505-494c-8081-7ef4c53b9a6f" (UID: "318faa10-7505-494c-8081-7ef4c53b9a6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:26:58.616372 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.616268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "318faa10-7505-494c-8081-7ef4c53b9a6f" (UID: "318faa10-7505-494c-8081-7ef4c53b9a6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:26:58.616474 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.616402    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-home" (OuterVolumeSpecName: "home") pod "318faa10-7505-494c-8081-7ef4c53b9a6f" (UID: "318faa10-7505-494c-8081-7ef4c53b9a6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:26:58.616574 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.616470    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:26:58.616574 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.616484    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:26:58.616574 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.616527    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:26:58.618331 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.618309    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-dshm" (OuterVolumeSpecName: "dshm") pod "318faa10-7505-494c-8081-7ef4c53b9a6f" (UID: "318faa10-7505-494c-8081-7ef4c53b9a6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:26:58.618440 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.618336    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/318faa10-7505-494c-8081-7ef4c53b9a6f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "318faa10-7505-494c-8081-7ef4c53b9a6f" (UID: "318faa10-7505-494c-8081-7ef4c53b9a6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:26:58.618440 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.618421    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318faa10-7505-494c-8081-7ef4c53b9a6f-kube-api-access-fcpvx" (OuterVolumeSpecName: "kube-api-access-fcpvx") pod "318faa10-7505-494c-8081-7ef4c53b9a6f" (UID: "318faa10-7505-494c-8081-7ef4c53b9a6f"). InnerVolumeSpecName "kube-api-access-fcpvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:26:58.717695 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.717660    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/318faa10-7505-494c-8081-7ef4c53b9a6f-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:26:58.717695 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.717688    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcpvx\" (UniqueName: \"kubernetes.io/projected/318faa10-7505-494c-8081-7ef4c53b9a6f-kube-api-access-fcpvx\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:26:58.717695 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.717700    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/318faa10-7505-494c-8081-7ef4c53b9a6f-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:26:58.885609 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.885574    2563 generic.go:356] "Generic (PLEG): container finished" podID="318faa10-7505-494c-8081-7ef4c53b9a6f" containerID="097d754a98973ff9ad607db65d0e81cdcc1a7a3b25bb27d0c6ac7a0ef8f9e088" exitCode=0
Sep 10 23:26:58.885762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.885620    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"
Sep 10 23:26:58.885762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.885662    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz" event={"ID":"318faa10-7505-494c-8081-7ef4c53b9a6f","Type":"ContainerDied","Data":"097d754a98973ff9ad607db65d0e81cdcc1a7a3b25bb27d0c6ac7a0ef8f9e088"}
Sep 10 23:26:58.885762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.885706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz" event={"ID":"318faa10-7505-494c-8081-7ef4c53b9a6f","Type":"ContainerDied","Data":"c8ca4807cda038c3419eaa9acf7bda9a8985c6d7bbbaca16e88b73fef2f4b593"}
Sep 10 23:26:58.885762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.885723    2563 scope.go:117] "RemoveContainer" containerID="097d754a98973ff9ad607db65d0e81cdcc1a7a3b25bb27d0c6ac7a0ef8f9e088"
Sep 10 23:26:58.895780 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.895761    2563 scope.go:117] "RemoveContainer" containerID="097d754a98973ff9ad607db65d0e81cdcc1a7a3b25bb27d0c6ac7a0ef8f9e088"
Sep 10 23:26:58.896023 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:26:58.896006    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097d754a98973ff9ad607db65d0e81cdcc1a7a3b25bb27d0c6ac7a0ef8f9e088\": container with ID starting with 097d754a98973ff9ad607db65d0e81cdcc1a7a3b25bb27d0c6ac7a0ef8f9e088 not found: ID does not exist" containerID="097d754a98973ff9ad607db65d0e81cdcc1a7a3b25bb27d0c6ac7a0ef8f9e088"
Sep 10 23:26:58.896059 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.896034    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097d754a98973ff9ad607db65d0e81cdcc1a7a3b25bb27d0c6ac7a0ef8f9e088"} err="failed to get container status \"097d754a98973ff9ad607db65d0e81cdcc1a7a3b25bb27d0c6ac7a0ef8f9e088\": rpc error: code = NotFound desc = could not find container \"097d754a98973ff9ad607db65d0e81cdcc1a7a3b25bb27d0c6ac7a0ef8f9e088\": container with ID starting with 097d754a98973ff9ad607db65d0e81cdcc1a7a3b25bb27d0c6ac7a0ef8f9e088 not found: ID does not exist"
Sep 10 23:26:58.906466 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.906443    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"]
Sep 10 23:26:58.910078 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:58.910057    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/e2e-conflict-b-d74b488b-kserve-76f5f6c66d-g7ppz"]
Sep 10 23:26:59.815061 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:26:59.815030    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="318faa10-7505-494c-8081-7ef4c53b9a6f" path="/var/lib/kubelet/pods/318faa10-7505-494c-8081-7ef4c53b9a6f/volumes"
Sep 10 23:27:08.112713 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.112676    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"]
Sep 10 23:27:08.113271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.112949    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj" podUID="ee993953-b6e4-456f-8a8a-fec7734e29fa" containerName="payload-processing" containerID="cri-o://14638fc6477970a5c976b3112326470ced09f7d0302f1d593d7b484d5d9aa9a7" gracePeriod=30
Sep 10 23:27:08.271293 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.271254    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5"]
Sep 10 23:27:08.274636 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.274609    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a4c77c-f2cb5"]
Sep 10 23:27:08.377924 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.377866    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"
Sep 10 23:27:08.512868 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.512831    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sr8z7\" (UniqueName: \"kubernetes.io/projected/ee993953-b6e4-456f-8a8a-fec7734e29fa-kube-api-access-sr8z7\") pod \"ee993953-b6e4-456f-8a8a-fec7734e29fa\" (UID: \"ee993953-b6e4-456f-8a8a-fec7734e29fa\") "
Sep 10 23:27:08.512868 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.512872    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee993953-b6e4-456f-8a8a-fec7734e29fa-plugins-config-volume\") pod \"ee993953-b6e4-456f-8a8a-fec7734e29fa\" (UID: \"ee993953-b6e4-456f-8a8a-fec7734e29fa\") "
Sep 10 23:27:08.513310 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.513281    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee993953-b6e4-456f-8a8a-fec7734e29fa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee993953-b6e4-456f-8a8a-fec7734e29fa" (UID: "ee993953-b6e4-456f-8a8a-fec7734e29fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:08.514984 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.514963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee993953-b6e4-456f-8a8a-fec7734e29fa-kube-api-access-sr8z7" (OuterVolumeSpecName: "kube-api-access-sr8z7") pod "ee993953-b6e4-456f-8a8a-fec7734e29fa" (UID: "ee993953-b6e4-456f-8a8a-fec7734e29fa"). InnerVolumeSpecName "kube-api-access-sr8z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:08.614215 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.614183    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr8z7\" (UniqueName: \"kubernetes.io/projected/ee993953-b6e4-456f-8a8a-fec7734e29fa-kube-api-access-sr8z7\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:27:08.614215 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.614210    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee993953-b6e4-456f-8a8a-fec7734e29fa-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:27:08.945753 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.945719    2563 generic.go:356] "Generic (PLEG): container finished" podID="ee993953-b6e4-456f-8a8a-fec7734e29fa" containerID="14638fc6477970a5c976b3112326470ced09f7d0302f1d593d7b484d5d9aa9a7" exitCode=0
Sep 10 23:27:08.945925 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.945775    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"
Sep 10 23:27:08.945925 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.945777    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj" event={"ID":"ee993953-b6e4-456f-8a8a-fec7734e29fa","Type":"ContainerDied","Data":"14638fc6477970a5c976b3112326470ced09f7d0302f1d593d7b484d5d9aa9a7"}
Sep 10 23:27:08.945925 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.945903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj" event={"ID":"ee993953-b6e4-456f-8a8a-fec7734e29fa","Type":"ContainerDied","Data":"b67e563c29c33b6ae8bbfa0f9a3bb3248dd3ace66f83d8dae76f7f14dcf97ebf"}
Sep 10 23:27:08.946028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.945933    2563 scope.go:117] "RemoveContainer" containerID="14638fc6477970a5c976b3112326470ced09f7d0302f1d593d7b484d5d9aa9a7"
Sep 10 23:27:08.955766 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.955751    2563 scope.go:117] "RemoveContainer" containerID="14638fc6477970a5c976b3112326470ced09f7d0302f1d593d7b484d5d9aa9a7"
Sep 10 23:27:08.956005 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:27:08.955986    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14638fc6477970a5c976b3112326470ced09f7d0302f1d593d7b484d5d9aa9a7\": container with ID starting with 14638fc6477970a5c976b3112326470ced09f7d0302f1d593d7b484d5d9aa9a7 not found: ID does not exist" containerID="14638fc6477970a5c976b3112326470ced09f7d0302f1d593d7b484d5d9aa9a7"
Sep 10 23:27:08.956057 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.956013    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14638fc6477970a5c976b3112326470ced09f7d0302f1d593d7b484d5d9aa9a7"} err="failed to get container status \"14638fc6477970a5c976b3112326470ced09f7d0302f1d593d7b484d5d9aa9a7\": rpc error: code = NotFound desc = could not find container \"14638fc6477970a5c976b3112326470ced09f7d0302f1d593d7b484d5d9aa9a7\": container with ID starting with 14638fc6477970a5c976b3112326470ced09f7d0302f1d593d7b484d5d9aa9a7 not found: ID does not exist"
Sep 10 23:27:08.966981 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.966956    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"]
Sep 10 23:27:08.970121 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:08.970101    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a4c77c-d4cc46797-qxvsj"]
Sep 10 23:27:09.814477 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:09.814443    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f" path="/var/lib/kubelet/pods/01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f/volumes"
Sep 10 23:27:09.814849 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:09.814836    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee993953-b6e4-456f-8a8a-fec7734e29fa" path="/var/lib/kubelet/pods/ee993953-b6e4-456f-8a8a-fec7734e29fa/volumes"
Sep 10 23:27:15.128934 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.128896    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j"]
Sep 10 23:27:15.129355 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.129343    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="318faa10-7505-494c-8081-7ef4c53b9a6f" containerName="main"
Sep 10 23:27:15.129355 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.129356    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="318faa10-7505-494c-8081-7ef4c53b9a6f" containerName="main"
Sep 10 23:27:15.129441 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.129372    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee993953-b6e4-456f-8a8a-fec7734e29fa" containerName="payload-processing"
Sep 10 23:27:15.129441 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.129378    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee993953-b6e4-456f-8a8a-fec7734e29fa" containerName="payload-processing"
Sep 10 23:27:15.129441 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.129394    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f" containerName="revoke-keys"
Sep 10 23:27:15.129441 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.129399    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f" containerName="revoke-keys"
Sep 10 23:27:15.129601 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.129462    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01d6157d-8f5c-4c6a-837b-1f3a0fd83b0f" containerName="revoke-keys"
Sep 10 23:27:15.129601 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.129470    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="318faa10-7505-494c-8081-7ef4c53b9a6f" containerName="main"
Sep 10 23:27:15.129601 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.129476    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee993953-b6e4-456f-8a8a-fec7734e29fa" containerName="payload-processing"
Sep 10 23:27:15.133849 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.133834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j"
Sep 10 23:27:15.136350 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.136324    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:27:15.136506 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.136415    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:27:15.140657 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.140632    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j"]
Sep 10 23:27:15.183043 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.183017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-72f076-9999j\" (UID: \"f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j"
Sep 10 23:27:15.284327 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.284284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-72f076-9999j\" (UID: \"f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j"
Sep 10 23:27:15.284900 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.284879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-72f076-9999j\" (UID: \"f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j"
Sep 10 23:27:15.445470 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.445397    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j"
Sep 10 23:27:15.715656 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.715626    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j"]
Sep 10 23:27:15.718157 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:27:15.718130    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22a0cd3_cae0_4c08_ae3b_0d0ff3d9e5bc.slice/crio-a461cdbe540c8bd00a1d42f08726040889de1d15d1f10848811d53c21b6c07dd WatchSource:0}: Error finding container a461cdbe540c8bd00a1d42f08726040889de1d15d1f10848811d53c21b6c07dd: Status 404 returned error can't find the container with id a461cdbe540c8bd00a1d42f08726040889de1d15d1f10848811d53c21b6c07dd
Sep 10 23:27:15.988325 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:15.988246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j" event={"ID":"f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc","Type":"ContainerStarted","Data":"a461cdbe540c8bd00a1d42f08726040889de1d15d1f10848811d53c21b6c07dd"}
Sep 10 23:27:16.998449 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:16.998417    2563 generic.go:356] "Generic (PLEG): container finished" podID="f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc" containerID="f8123ff4f11754f1e9807669ffc56f9568d14a08bd5ae44f4c47ba671bd0a2d7" exitCode=0
Sep 10 23:27:16.998837 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:16.998462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j" event={"ID":"f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc","Type":"ContainerDied","Data":"f8123ff4f11754f1e9807669ffc56f9568d14a08bd5ae44f4c47ba671bd0a2d7"}
Sep 10 23:27:18.148005 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:18.147982    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j"
Sep 10 23:27:18.215149 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:18.215118    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc-maas-api-service-ca\") pod \"f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc\" (UID: \"f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc\") "
Sep 10 23:27:18.215460 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:18.215438    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc" (UID: "f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:18.316676 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:18.316647    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:27:19.011354 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:19.011322    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j"
Sep 10 23:27:19.011549 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:19.011357    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j" event={"ID":"f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc","Type":"ContainerDied","Data":"a461cdbe540c8bd00a1d42f08726040889de1d15d1f10848811d53c21b6c07dd"}
Sep 10 23:27:19.011549 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:19.011404    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a461cdbe540c8bd00a1d42f08726040889de1d15d1f10848811d53c21b6c07dd"
Sep 10 23:27:20.131457 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:20.131427    2563 scope.go:117] "RemoveContainer" containerID="be26dea0588c1a9aafdf46a61051cf79fe3a60d1d45b36b9c085c4c50f5a4585"
Sep 10 23:27:30.712095 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:30.712062    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"]
Sep 10 23:27:30.712601 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:30.712386    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm" podUID="10900502-db8a-4c8e-9354-0cb50b786f0e" containerName="payload-pre-processing" containerID="cri-o://a62196f828f07a91bd26ac147db86ad900c98508a10b63141d5e162c35ba8dfe" gracePeriod=30
Sep 10 23:27:30.858087 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:30.858050    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j"]
Sep 10 23:27:30.861406 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:30.861375    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-72f076-9999j"]
Sep 10 23:27:30.974838 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:30.974783    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"
Sep 10 23:27:31.034564 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.034540    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10900502-db8a-4c8e-9354-0cb50b786f0e-plugins-config-volume\") pod \"10900502-db8a-4c8e-9354-0cb50b786f0e\" (UID: \"10900502-db8a-4c8e-9354-0cb50b786f0e\") "
Sep 10 23:27:31.034700 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.034599    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-852sz\" (UniqueName: \"kubernetes.io/projected/10900502-db8a-4c8e-9354-0cb50b786f0e-kube-api-access-852sz\") pod \"10900502-db8a-4c8e-9354-0cb50b786f0e\" (UID: \"10900502-db8a-4c8e-9354-0cb50b786f0e\") "
Sep 10 23:27:31.034916 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.034894    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10900502-db8a-4c8e-9354-0cb50b786f0e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "10900502-db8a-4c8e-9354-0cb50b786f0e" (UID: "10900502-db8a-4c8e-9354-0cb50b786f0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:31.035020 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.035005    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10900502-db8a-4c8e-9354-0cb50b786f0e-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:27:31.036559 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.036541    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10900502-db8a-4c8e-9354-0cb50b786f0e-kube-api-access-852sz" (OuterVolumeSpecName: "kube-api-access-852sz") pod "10900502-db8a-4c8e-9354-0cb50b786f0e" (UID: "10900502-db8a-4c8e-9354-0cb50b786f0e"). InnerVolumeSpecName "kube-api-access-852sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:31.076651 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.076622    2563 generic.go:356] "Generic (PLEG): container finished" podID="10900502-db8a-4c8e-9354-0cb50b786f0e" containerID="a62196f828f07a91bd26ac147db86ad900c98508a10b63141d5e162c35ba8dfe" exitCode=0
Sep 10 23:27:31.076758 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.076662    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"
Sep 10 23:27:31.076758 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.076701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm" event={"ID":"10900502-db8a-4c8e-9354-0cb50b786f0e","Type":"ContainerDied","Data":"a62196f828f07a91bd26ac147db86ad900c98508a10b63141d5e162c35ba8dfe"}
Sep 10 23:27:31.076841 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.076757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm" event={"ID":"10900502-db8a-4c8e-9354-0cb50b786f0e","Type":"ContainerDied","Data":"86f3bb79bf63c482e22e2bd106fd23fd672d94934a85db68733682f872613798"}
Sep 10 23:27:31.076841 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.076788    2563 scope.go:117] "RemoveContainer" containerID="a62196f828f07a91bd26ac147db86ad900c98508a10b63141d5e162c35ba8dfe"
Sep 10 23:27:31.086751 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.086735    2563 scope.go:117] "RemoveContainer" containerID="a62196f828f07a91bd26ac147db86ad900c98508a10b63141d5e162c35ba8dfe"
Sep 10 23:27:31.087004 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:27:31.086988    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a62196f828f07a91bd26ac147db86ad900c98508a10b63141d5e162c35ba8dfe\": container with ID starting with a62196f828f07a91bd26ac147db86ad900c98508a10b63141d5e162c35ba8dfe not found: ID does not exist" containerID="a62196f828f07a91bd26ac147db86ad900c98508a10b63141d5e162c35ba8dfe"
Sep 10 23:27:31.087055 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.087012    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62196f828f07a91bd26ac147db86ad900c98508a10b63141d5e162c35ba8dfe"} err="failed to get container status \"a62196f828f07a91bd26ac147db86ad900c98508a10b63141d5e162c35ba8dfe\": rpc error: code = NotFound desc = could not find container \"a62196f828f07a91bd26ac147db86ad900c98508a10b63141d5e162c35ba8dfe\": container with ID starting with a62196f828f07a91bd26ac147db86ad900c98508a10b63141d5e162c35ba8dfe not found: ID does not exist"
Sep 10 23:27:31.097277 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.097248    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"]
Sep 10 23:27:31.100680 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.100661    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-72f076-6d5866f554-cjcjm"]
Sep 10 23:27:31.136134 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.136114    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-852sz\" (UniqueName: \"kubernetes.io/projected/10900502-db8a-4c8e-9354-0cb50b786f0e-kube-api-access-852sz\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:27:31.815786 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.815753    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10900502-db8a-4c8e-9354-0cb50b786f0e" path="/var/lib/kubelet/pods/10900502-db8a-4c8e-9354-0cb50b786f0e/volumes"
Sep 10 23:27:31.816282 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:31.816260    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc" path="/var/lib/kubelet/pods/f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc/volumes"
Sep 10 23:27:37.562962 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.562927    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp"]
Sep 10 23:27:37.563371 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.563357    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc" containerName="revoke-keys"
Sep 10 23:27:37.563371 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.563371    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc" containerName="revoke-keys"
Sep 10 23:27:37.563466 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.563392    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10900502-db8a-4c8e-9354-0cb50b786f0e" containerName="payload-pre-processing"
Sep 10 23:27:37.563466 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.563398    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="10900502-db8a-4c8e-9354-0cb50b786f0e" containerName="payload-pre-processing"
Sep 10 23:27:37.563466 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.563460    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="10900502-db8a-4c8e-9354-0cb50b786f0e" containerName="payload-pre-processing"
Sep 10 23:27:37.563590 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.563472    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f22a0cd3-cae0-4c08-ae3b-0d0ff3d9e5bc" containerName="revoke-keys"
Sep 10 23:27:37.568327 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.568305    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp"
Sep 10 23:27:37.572116 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.572090    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:27:37.572241 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.572092    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:27:37.574454 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.574430    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp"]
Sep 10 23:27:37.702147 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.702118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167424d-9d0c-4b15-8485-8dd3f51dec11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp\" (UID: \"7167424d-9d0c-4b15-8485-8dd3f51dec11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp"
Sep 10 23:27:37.803376 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.803345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167424d-9d0c-4b15-8485-8dd3f51dec11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp\" (UID: \"7167424d-9d0c-4b15-8485-8dd3f51dec11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp"
Sep 10 23:27:37.803987 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.803967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167424d-9d0c-4b15-8485-8dd3f51dec11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp\" (UID: \"7167424d-9d0c-4b15-8485-8dd3f51dec11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp"
Sep 10 23:27:37.880435 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:37.880369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp"
Sep 10 23:27:38.152349 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:38.152325    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp"]
Sep 10 23:27:38.154298 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:27:38.154267    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7167424d_9d0c_4b15_8485_8dd3f51dec11.slice/crio-2d016090a8119a8c73deb48f7a09b8e7962e94c9265e1fbdb75e9f8059170969 WatchSource:0}: Error finding container 2d016090a8119a8c73deb48f7a09b8e7962e94c9265e1fbdb75e9f8059170969: Status 404 returned error can't find the container with id 2d016090a8119a8c73deb48f7a09b8e7962e94c9265e1fbdb75e9f8059170969
Sep 10 23:27:39.124974 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:39.124883    2563 generic.go:356] "Generic (PLEG): container finished" podID="7167424d-9d0c-4b15-8485-8dd3f51dec11" containerID="ccefcac3f0518569822b0aa9d68bb2d12016f16b1740d8200c2c675cf97b31fa" exitCode=0
Sep 10 23:27:39.124974 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:39.124933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp" event={"ID":"7167424d-9d0c-4b15-8485-8dd3f51dec11","Type":"ContainerDied","Data":"ccefcac3f0518569822b0aa9d68bb2d12016f16b1740d8200c2c675cf97b31fa"}
Sep 10 23:27:39.124974 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:39.124974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp" event={"ID":"7167424d-9d0c-4b15-8485-8dd3f51dec11","Type":"ContainerStarted","Data":"2d016090a8119a8c73deb48f7a09b8e7962e94c9265e1fbdb75e9f8059170969"}
Sep 10 23:27:40.267190 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:40.267167    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp"
Sep 10 23:27:40.431256 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:40.431172    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167424d-9d0c-4b15-8485-8dd3f51dec11-maas-api-service-ca\") pod \"7167424d-9d0c-4b15-8485-8dd3f51dec11\" (UID: \"7167424d-9d0c-4b15-8485-8dd3f51dec11\") "
Sep 10 23:27:40.431546 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:40.431525    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7167424d-9d0c-4b15-8485-8dd3f51dec11-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7167424d-9d0c-4b15-8485-8dd3f51dec11" (UID: "7167424d-9d0c-4b15-8485-8dd3f51dec11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:40.532167 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:40.532129    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7167424d-9d0c-4b15-8485-8dd3f51dec11-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:27:41.137065 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:41.137039    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp"
Sep 10 23:27:41.137232 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:41.137071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp" event={"ID":"7167424d-9d0c-4b15-8485-8dd3f51dec11","Type":"ContainerDied","Data":"2d016090a8119a8c73deb48f7a09b8e7962e94c9265e1fbdb75e9f8059170969"}
Sep 10 23:27:41.137232 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:41.137113    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d016090a8119a8c73deb48f7a09b8e7962e94c9265e1fbdb75e9f8059170969"
Sep 10 23:27:48.380255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:48.380208    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp"]
Sep 10 23:27:48.385514 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:48.384961    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-bda598-54xhp"]
Sep 10 23:27:49.814371 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:49.814334    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7167424d-9d0c-4b15-8485-8dd3f51dec11" path="/var/lib/kubelet/pods/7167424d-9d0c-4b15-8485-8dd3f51dec11/volumes"
Sep 10 23:27:55.381370 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:55.381331    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg"]
Sep 10 23:27:55.382004 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:55.381983    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7167424d-9d0c-4b15-8485-8dd3f51dec11" containerName="revoke-keys"
Sep 10 23:27:55.382103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:55.382006    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7167424d-9d0c-4b15-8485-8dd3f51dec11" containerName="revoke-keys"
Sep 10 23:27:55.382103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:55.382096    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7167424d-9d0c-4b15-8485-8dd3f51dec11" containerName="revoke-keys"
Sep 10 23:27:55.385286 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:55.385263    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg"
Sep 10 23:27:55.387478 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:55.387455    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:27:55.387612 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:55.387564    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:27:55.393667 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:55.393643    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg"]
Sep 10 23:27:55.473047 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:55.473006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2863c05d-3210-4042-b4eb-4382ea314468-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg\" (UID: \"2863c05d-3210-4042-b4eb-4382ea314468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg"
Sep 10 23:27:55.573950 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:55.573920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2863c05d-3210-4042-b4eb-4382ea314468-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg\" (UID: \"2863c05d-3210-4042-b4eb-4382ea314468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg"
Sep 10 23:27:55.574574 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:55.574556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2863c05d-3210-4042-b4eb-4382ea314468-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg\" (UID: \"2863c05d-3210-4042-b4eb-4382ea314468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg"
Sep 10 23:27:55.697019 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:55.696929    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg"
Sep 10 23:27:55.973680 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:55.973643    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg"]
Sep 10 23:27:55.975061 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:27:55.975032    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2863c05d_3210_4042_b4eb_4382ea314468.slice/crio-94fbf3efe4c25f0703dc5780e015e9412b3b2aacb1a7b6a4ddd49dc17d5a43ac WatchSource:0}: Error finding container 94fbf3efe4c25f0703dc5780e015e9412b3b2aacb1a7b6a4ddd49dc17d5a43ac: Status 404 returned error can't find the container with id 94fbf3efe4c25f0703dc5780e015e9412b3b2aacb1a7b6a4ddd49dc17d5a43ac
Sep 10 23:27:56.220552 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:56.220512    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg" event={"ID":"2863c05d-3210-4042-b4eb-4382ea314468","Type":"ContainerStarted","Data":"2e49c0e8bad8125d9d2d901963fa42534e5ca832830d2eb99885383028763d2a"}
Sep 10 23:27:56.220750 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:56.220576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg" event={"ID":"2863c05d-3210-4042-b4eb-4382ea314468","Type":"ContainerStarted","Data":"94fbf3efe4c25f0703dc5780e015e9412b3b2aacb1a7b6a4ddd49dc17d5a43ac"}
Sep 10 23:27:56.238515 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:56.238398    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg" podStartSLOduration=1.238382786 podStartE2EDuration="1.238382786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:27:56.236306924 +0000 UTC m=+1596.927884134" watchObservedRunningTime="2026-09-10 23:27:56.238382786 +0000 UTC m=+1596.929959968"
Sep 10 23:27:57.228058 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:57.228019    2563 generic.go:356] "Generic (PLEG): container finished" podID="2863c05d-3210-4042-b4eb-4382ea314468" containerID="2e49c0e8bad8125d9d2d901963fa42534e5ca832830d2eb99885383028763d2a" exitCode=0
Sep 10 23:27:57.228542 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:57.228071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg" event={"ID":"2863c05d-3210-4042-b4eb-4382ea314468","Type":"ContainerDied","Data":"2e49c0e8bad8125d9d2d901963fa42534e5ca832830d2eb99885383028763d2a"}
Sep 10 23:27:58.365588 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:58.365565    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg"
Sep 10 23:27:58.401640 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:58.401603    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2863c05d-3210-4042-b4eb-4382ea314468-maas-api-service-ca\") pod \"2863c05d-3210-4042-b4eb-4382ea314468\" (UID: \"2863c05d-3210-4042-b4eb-4382ea314468\") "
Sep 10 23:27:58.401927 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:58.401902    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2863c05d-3210-4042-b4eb-4382ea314468-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2863c05d-3210-4042-b4eb-4382ea314468" (UID: "2863c05d-3210-4042-b4eb-4382ea314468"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:58.502473 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:58.502386    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2863c05d-3210-4042-b4eb-4382ea314468-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:27:59.240098 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:59.240070    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg"
Sep 10 23:27:59.240270 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:59.240106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg" event={"ID":"2863c05d-3210-4042-b4eb-4382ea314468","Type":"ContainerDied","Data":"94fbf3efe4c25f0703dc5780e015e9412b3b2aacb1a7b6a4ddd49dc17d5a43ac"}
Sep 10 23:27:59.240270 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:27:59.240154    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94fbf3efe4c25f0703dc5780e015e9412b3b2aacb1a7b6a4ddd49dc17d5a43ac"
Sep 10 23:28:06.179949 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:06.179909    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"]
Sep 10 23:28:06.180417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:06.180142    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f" podUID="5484738f-b75c-4604-99b6-90819dd32d96" containerName="payload-pre-processing" containerID="cri-o://804889ee87d19fadc4e5b23083da7a6c76fe99a0f2c52fbb45d408535da3e2fc" gracePeriod=30
Sep 10 23:28:06.428965 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:06.428941    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"
Sep 10 23:28:06.490235 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:06.490161    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5484738f-b75c-4604-99b6-90819dd32d96-plugins-config-volume\") pod \"5484738f-b75c-4604-99b6-90819dd32d96\" (UID: \"5484738f-b75c-4604-99b6-90819dd32d96\") "
Sep 10 23:28:06.490235 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:06.490234    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmlpb\" (UniqueName: \"kubernetes.io/projected/5484738f-b75c-4604-99b6-90819dd32d96-kube-api-access-tmlpb\") pod \"5484738f-b75c-4604-99b6-90819dd32d96\" (UID: \"5484738f-b75c-4604-99b6-90819dd32d96\") "
Sep 10 23:28:06.490649 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:06.490624    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5484738f-b75c-4604-99b6-90819dd32d96-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5484738f-b75c-4604-99b6-90819dd32d96" (UID: "5484738f-b75c-4604-99b6-90819dd32d96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:28:06.492361 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:06.492338    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5484738f-b75c-4604-99b6-90819dd32d96-kube-api-access-tmlpb" (OuterVolumeSpecName: "kube-api-access-tmlpb") pod "5484738f-b75c-4604-99b6-90819dd32d96" (UID: "5484738f-b75c-4604-99b6-90819dd32d96"). InnerVolumeSpecName "kube-api-access-tmlpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:28:06.591732 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:06.591695    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5484738f-b75c-4604-99b6-90819dd32d96-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:28:06.591732 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:06.591727    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmlpb\" (UniqueName: \"kubernetes.io/projected/5484738f-b75c-4604-99b6-90819dd32d96-kube-api-access-tmlpb\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:28:06.893292 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:06.893257    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg"]
Sep 10 23:28:06.897407 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:06.897378    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-7b202f-9qtsg"]
Sep 10 23:28:07.285289 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:07.285194    2563 generic.go:356] "Generic (PLEG): container finished" podID="5484738f-b75c-4604-99b6-90819dd32d96" containerID="804889ee87d19fadc4e5b23083da7a6c76fe99a0f2c52fbb45d408535da3e2fc" exitCode=0
Sep 10 23:28:07.285289 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:07.285237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f" event={"ID":"5484738f-b75c-4604-99b6-90819dd32d96","Type":"ContainerDied","Data":"804889ee87d19fadc4e5b23083da7a6c76fe99a0f2c52fbb45d408535da3e2fc"}
Sep 10 23:28:07.285289 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:07.285248    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"
Sep 10 23:28:07.285289 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:07.285290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f" event={"ID":"5484738f-b75c-4604-99b6-90819dd32d96","Type":"ContainerDied","Data":"65ef108ab441676428306736674c39340beb0ab1e674deb4695bdec0514c7b04"}
Sep 10 23:28:07.285967 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:07.285313    2563 scope.go:117] "RemoveContainer" containerID="804889ee87d19fadc4e5b23083da7a6c76fe99a0f2c52fbb45d408535da3e2fc"
Sep 10 23:28:07.297151 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:07.297106    2563 scope.go:117] "RemoveContainer" containerID="804889ee87d19fadc4e5b23083da7a6c76fe99a0f2c52fbb45d408535da3e2fc"
Sep 10 23:28:07.297555 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:28:07.297534    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804889ee87d19fadc4e5b23083da7a6c76fe99a0f2c52fbb45d408535da3e2fc\": container with ID starting with 804889ee87d19fadc4e5b23083da7a6c76fe99a0f2c52fbb45d408535da3e2fc not found: ID does not exist" containerID="804889ee87d19fadc4e5b23083da7a6c76fe99a0f2c52fbb45d408535da3e2fc"
Sep 10 23:28:07.297638 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:07.297569    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804889ee87d19fadc4e5b23083da7a6c76fe99a0f2c52fbb45d408535da3e2fc"} err="failed to get container status \"804889ee87d19fadc4e5b23083da7a6c76fe99a0f2c52fbb45d408535da3e2fc\": rpc error: code = NotFound desc = could not find container \"804889ee87d19fadc4e5b23083da7a6c76fe99a0f2c52fbb45d408535da3e2fc\": container with ID starting with 804889ee87d19fadc4e5b23083da7a6c76fe99a0f2c52fbb45d408535da3e2fc not found: ID does not exist"
Sep 10 23:28:07.308885 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:07.308859    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"]
Sep 10 23:28:07.312014 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:07.311992    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7b202f-77c5474c46-r2x7f"]
Sep 10 23:28:07.814648 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:07.814614    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2863c05d-3210-4042-b4eb-4382ea314468" path="/var/lib/kubelet/pods/2863c05d-3210-4042-b4eb-4382ea314468/volumes"
Sep 10 23:28:07.815026 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:07.815009    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5484738f-b75c-4604-99b6-90819dd32d96" path="/var/lib/kubelet/pods/5484738f-b75c-4604-99b6-90819dd32d96/volumes"
Sep 10 23:28:20.163462 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:20.163389    2563 scope.go:117] "RemoveContainer" containerID="09d0c78e4122923de9db3bc49195250208ab26a0039e54aca095c6191f8ec1d4"
Sep 10 23:28:20.171956 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:20.171935    2563 scope.go:117] "RemoveContainer" containerID="e7054d81da24f8ecec53862c9863ca9d47d372a756c7d11330ab917be4922dad"
Sep 10 23:28:21.656234 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:21.656200    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"]
Sep 10 23:28:36.675645 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.675594    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk" podUID="05ddd8f1-de97-4ad9-be78-e68f04c0e88b" containerName="main" containerID="cri-o://69c8023f15be0552ff5fef7a89ce62ccf99defe717b03422dd3f02fc89982962" gracePeriod=45
Sep 10 23:28:36.929052 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.928996    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:28:36.984867 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.984838    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-home\") pod \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") "
Sep 10 23:28:36.985029 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.984891    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-model-cache\") pod \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") "
Sep 10 23:28:36.985029 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.984909    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-tmp-dir\") pod \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") "
Sep 10 23:28:36.985029 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.984970    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk7nh\" (UniqueName: \"kubernetes.io/projected/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-kube-api-access-gk7nh\") pod \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") "
Sep 10 23:28:36.985029 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.984998    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-dshm\") pod \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") "
Sep 10 23:28:36.985029 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.985014    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-tls-certs\") pod \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\" (UID: \"05ddd8f1-de97-4ad9-be78-e68f04c0e88b\") "
Sep 10 23:28:36.985285 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.985122    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-home" (OuterVolumeSpecName: "home") pod "05ddd8f1-de97-4ad9-be78-e68f04c0e88b" (UID: "05ddd8f1-de97-4ad9-be78-e68f04c0e88b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:36.985285 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.985162    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-model-cache" (OuterVolumeSpecName: "model-cache") pod "05ddd8f1-de97-4ad9-be78-e68f04c0e88b" (UID: "05ddd8f1-de97-4ad9-be78-e68f04c0e88b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:36.985285 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.985209    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05ddd8f1-de97-4ad9-be78-e68f04c0e88b" (UID: "05ddd8f1-de97-4ad9-be78-e68f04c0e88b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:36.985417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.985364    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:28:36.985417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.985377    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:28:36.985417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.985387    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:28:36.987266 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.987240    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05ddd8f1-de97-4ad9-be78-e68f04c0e88b" (UID: "05ddd8f1-de97-4ad9-be78-e68f04c0e88b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:28:36.987266 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.987245    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-kube-api-access-gk7nh" (OuterVolumeSpecName: "kube-api-access-gk7nh") pod "05ddd8f1-de97-4ad9-be78-e68f04c0e88b" (UID: "05ddd8f1-de97-4ad9-be78-e68f04c0e88b"). InnerVolumeSpecName "kube-api-access-gk7nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:28:36.987405 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:36.987251    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-dshm" (OuterVolumeSpecName: "dshm") pod "05ddd8f1-de97-4ad9-be78-e68f04c0e88b" (UID: "05ddd8f1-de97-4ad9-be78-e68f04c0e88b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:28:37.086517 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:37.086464    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk7nh\" (UniqueName: \"kubernetes.io/projected/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-kube-api-access-gk7nh\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:28:37.086517 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:37.086520    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:28:37.086709 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:37.086536    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05ddd8f1-de97-4ad9-be78-e68f04c0e88b-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:28:37.454410 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:37.454376    2563 generic.go:356] "Generic (PLEG): container finished" podID="05ddd8f1-de97-4ad9-be78-e68f04c0e88b" containerID="69c8023f15be0552ff5fef7a89ce62ccf99defe717b03422dd3f02fc89982962" exitCode=0
Sep 10 23:28:37.454591 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:37.454416    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"
Sep 10 23:28:37.454591 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:37.454460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk" event={"ID":"05ddd8f1-de97-4ad9-be78-e68f04c0e88b","Type":"ContainerDied","Data":"69c8023f15be0552ff5fef7a89ce62ccf99defe717b03422dd3f02fc89982962"}
Sep 10 23:28:37.454591 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:37.454518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk" event={"ID":"05ddd8f1-de97-4ad9-be78-e68f04c0e88b","Type":"ContainerDied","Data":"90109ddaa3518c230fb07e7ef289a6376f98cd9ca1d1c63bce21af649dbc989b"}
Sep 10 23:28:37.454591 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:37.454538    2563 scope.go:117] "RemoveContainer" containerID="69c8023f15be0552ff5fef7a89ce62ccf99defe717b03422dd3f02fc89982962"
Sep 10 23:28:37.464086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:37.464066    2563 scope.go:117] "RemoveContainer" containerID="69c8023f15be0552ff5fef7a89ce62ccf99defe717b03422dd3f02fc89982962"
Sep 10 23:28:37.464365 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:28:37.464345    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c8023f15be0552ff5fef7a89ce62ccf99defe717b03422dd3f02fc89982962\": container with ID starting with 69c8023f15be0552ff5fef7a89ce62ccf99defe717b03422dd3f02fc89982962 not found: ID does not exist" containerID="69c8023f15be0552ff5fef7a89ce62ccf99defe717b03422dd3f02fc89982962"
Sep 10 23:28:37.464444 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:37.464375    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c8023f15be0552ff5fef7a89ce62ccf99defe717b03422dd3f02fc89982962"} err="failed to get container status \"69c8023f15be0552ff5fef7a89ce62ccf99defe717b03422dd3f02fc89982962\": rpc error: code = NotFound desc = could not find container \"69c8023f15be0552ff5fef7a89ce62ccf99defe717b03422dd3f02fc89982962\": container with ID starting with 69c8023f15be0552ff5fef7a89ce62ccf99defe717b03422dd3f02fc89982962 not found: ID does not exist"
Sep 10 23:28:37.473965 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:37.473942    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"]
Sep 10 23:28:37.477237 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:37.477217    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/auth-test-model-6c97f2-kserve-86b9bb4b99-9zskk"]
Sep 10 23:28:37.813899 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:37.813871    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05ddd8f1-de97-4ad9-be78-e68f04c0e88b" path="/var/lib/kubelet/pods/05ddd8f1-de97-4ad9-be78-e68f04c0e88b/volumes"
Sep 10 23:28:47.704296 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.704259    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8"]
Sep 10 23:28:47.704703 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.704690    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2863c05d-3210-4042-b4eb-4382ea314468" containerName="revoke-keys"
Sep 10 23:28:47.704703 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.704704    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2863c05d-3210-4042-b4eb-4382ea314468" containerName="revoke-keys"
Sep 10 23:28:47.704780 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.704716    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5484738f-b75c-4604-99b6-90819dd32d96" containerName="payload-pre-processing"
Sep 10 23:28:47.704780 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.704722    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5484738f-b75c-4604-99b6-90819dd32d96" containerName="payload-pre-processing"
Sep 10 23:28:47.704780 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.704737    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05ddd8f1-de97-4ad9-be78-e68f04c0e88b" containerName="main"
Sep 10 23:28:47.704780 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.704743    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="05ddd8f1-de97-4ad9-be78-e68f04c0e88b" containerName="main"
Sep 10 23:28:47.704914 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.704801    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5484738f-b75c-4604-99b6-90819dd32d96" containerName="payload-pre-processing"
Sep 10 23:28:47.704914 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.704811    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="05ddd8f1-de97-4ad9-be78-e68f04c0e88b" containerName="main"
Sep 10 23:28:47.704914 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.704818    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2863c05d-3210-4042-b4eb-4382ea314468" containerName="revoke-keys"
Sep 10 23:28:47.709150 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.709134    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8"
Sep 10 23:28:47.711478 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.711379    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:28:47.711478 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.711427    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:28:47.715481 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.715457    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8"]
Sep 10 23:28:47.791086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.791039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84598834-e1b5-490e-b44f-7dcdd2344ca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8\" (UID: \"84598834-e1b5-490e-b44f-7dcdd2344ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8"
Sep 10 23:28:47.892516 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.892458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84598834-e1b5-490e-b44f-7dcdd2344ca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8\" (UID: \"84598834-e1b5-490e-b44f-7dcdd2344ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8"
Sep 10 23:28:47.893094 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:47.893075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84598834-e1b5-490e-b44f-7dcdd2344ca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8\" (UID: \"84598834-e1b5-490e-b44f-7dcdd2344ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8"
Sep 10 23:28:48.022129 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:48.022040    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8"
Sep 10 23:28:48.291560 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:48.291470    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8"]
Sep 10 23:28:48.294828 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:28:48.294789    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84598834_e1b5_490e_b44f_7dcdd2344ca9.slice/crio-362b6ba60975371a87b2e78ed44e75dfe5f9bcb2d81e506a8641d74fdab0f124 WatchSource:0}: Error finding container 362b6ba60975371a87b2e78ed44e75dfe5f9bcb2d81e506a8641d74fdab0f124: Status 404 returned error can't find the container with id 362b6ba60975371a87b2e78ed44e75dfe5f9bcb2d81e506a8641d74fdab0f124
Sep 10 23:28:48.516278 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:48.516244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8" event={"ID":"84598834-e1b5-490e-b44f-7dcdd2344ca9","Type":"ContainerStarted","Data":"362b6ba60975371a87b2e78ed44e75dfe5f9bcb2d81e506a8641d74fdab0f124"}
Sep 10 23:28:49.523156 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:49.523120    2563 generic.go:356] "Generic (PLEG): container finished" podID="84598834-e1b5-490e-b44f-7dcdd2344ca9" containerID="1c337b5eb128e5ade6f73abba489d79f666736455990fd81f31c58c16bed4c6c" exitCode=0
Sep 10 23:28:49.523601 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:49.523175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8" event={"ID":"84598834-e1b5-490e-b44f-7dcdd2344ca9","Type":"ContainerDied","Data":"1c337b5eb128e5ade6f73abba489d79f666736455990fd81f31c58c16bed4c6c"}
Sep 10 23:28:50.664182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:50.664158    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8"
Sep 10 23:28:50.821779 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:50.821750    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84598834-e1b5-490e-b44f-7dcdd2344ca9-maas-api-service-ca\") pod \"84598834-e1b5-490e-b44f-7dcdd2344ca9\" (UID: \"84598834-e1b5-490e-b44f-7dcdd2344ca9\") "
Sep 10 23:28:50.822120 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:50.822097    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84598834-e1b5-490e-b44f-7dcdd2344ca9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "84598834-e1b5-490e-b44f-7dcdd2344ca9" (UID: "84598834-e1b5-490e-b44f-7dcdd2344ca9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:28:50.923354 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:50.923320    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84598834-e1b5-490e-b44f-7dcdd2344ca9-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:28:51.535229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:51.535198    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8"
Sep 10 23:28:51.535229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:51.535221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8" event={"ID":"84598834-e1b5-490e-b44f-7dcdd2344ca9","Type":"ContainerDied","Data":"362b6ba60975371a87b2e78ed44e75dfe5f9bcb2d81e506a8641d74fdab0f124"}
Sep 10 23:28:51.535645 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:51.535265    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="362b6ba60975371a87b2e78ed44e75dfe5f9bcb2d81e506a8641d74fdab0f124"
Sep 10 23:28:52.574959 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:52.574921    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"]
Sep 10 23:28:52.579606 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:52.579578    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"]
Sep 10 23:28:52.589370 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:52.589342    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"]
Sep 10 23:28:52.597385 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:52.597351    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"]
Sep 10 23:28:52.606145 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:28:52.606113    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"]
Sep 10 23:29:03.247292 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.247251    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7025b9-29818035-jhwml"]
Sep 10 23:29:03.247946 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.247818    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7025b9-29818035-jhwml"]
Sep 10 23:29:03.267845 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.267810    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"]
Sep 10 23:29:03.268711 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.268672    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv" podUID="5541bf40-af13-42a3-89d3-0c79e2a2b35b" containerName="payload-processing" containerID="cri-o://c13a844db1f9139f7a945c75d20d9997a276ce606e094cb1b22600e4ca62b2cf" gracePeriod=30
Sep 10 23:29:03.275844 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.275811    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"]
Sep 10 23:29:03.276096 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.276045    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7" podUID="f4084d5e-45e6-4acf-93d5-a9a9f2150b67" containerName="payload-pre-processing" containerID="cri-o://e14fb6d58c1174d532c8028059a6b033d0f2c992ea0115f22556fbcea1ec0adf" gracePeriod=30
Sep 10 23:29:03.433345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.433315    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8"]
Sep 10 23:29:03.437352 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.437321    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7025b9-srgr8"]
Sep 10 23:29:03.568879 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.568855    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"
Sep 10 23:29:03.572958 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.572934    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"
Sep 10 23:29:03.605831 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.605775    2563 generic.go:356] "Generic (PLEG): container finished" podID="5541bf40-af13-42a3-89d3-0c79e2a2b35b" containerID="c13a844db1f9139f7a945c75d20d9997a276ce606e094cb1b22600e4ca62b2cf" exitCode=0
Sep 10 23:29:03.605831 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.605818    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"
Sep 10 23:29:03.606049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.605846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv" event={"ID":"5541bf40-af13-42a3-89d3-0c79e2a2b35b","Type":"ContainerDied","Data":"c13a844db1f9139f7a945c75d20d9997a276ce606e094cb1b22600e4ca62b2cf"}
Sep 10 23:29:03.606049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.605901    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv" event={"ID":"5541bf40-af13-42a3-89d3-0c79e2a2b35b","Type":"ContainerDied","Data":"6a56610e4f546eabd127c4ab459edd5988d23271ff7cc7a030cbdf2182c97bae"}
Sep 10 23:29:03.606049 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.605929    2563 scope.go:117] "RemoveContainer" containerID="c13a844db1f9139f7a945c75d20d9997a276ce606e094cb1b22600e4ca62b2cf"
Sep 10 23:29:03.607452 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.607427    2563 generic.go:356] "Generic (PLEG): container finished" podID="f4084d5e-45e6-4acf-93d5-a9a9f2150b67" containerID="e14fb6d58c1174d532c8028059a6b033d0f2c992ea0115f22556fbcea1ec0adf" exitCode=0
Sep 10 23:29:03.607611 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.607592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7" event={"ID":"f4084d5e-45e6-4acf-93d5-a9a9f2150b67","Type":"ContainerDied","Data":"e14fb6d58c1174d532c8028059a6b033d0f2c992ea0115f22556fbcea1ec0adf"}
Sep 10 23:29:03.607698 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.607639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7" event={"ID":"f4084d5e-45e6-4acf-93d5-a9a9f2150b67","Type":"ContainerDied","Data":"d4457ce58646953bf432fc15e49a32d66fc1c59b0abd633bafd4a97bf92c60ca"}
Sep 10 23:29:03.607771 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.607708    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"
Sep 10 23:29:03.618120 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.618097    2563 scope.go:117] "RemoveContainer" containerID="c13a844db1f9139f7a945c75d20d9997a276ce606e094cb1b22600e4ca62b2cf"
Sep 10 23:29:03.618502 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:29:03.618470    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13a844db1f9139f7a945c75d20d9997a276ce606e094cb1b22600e4ca62b2cf\": container with ID starting with c13a844db1f9139f7a945c75d20d9997a276ce606e094cb1b22600e4ca62b2cf not found: ID does not exist" containerID="c13a844db1f9139f7a945c75d20d9997a276ce606e094cb1b22600e4ca62b2cf"
Sep 10 23:29:03.618591 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.618518    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13a844db1f9139f7a945c75d20d9997a276ce606e094cb1b22600e4ca62b2cf"} err="failed to get container status \"c13a844db1f9139f7a945c75d20d9997a276ce606e094cb1b22600e4ca62b2cf\": rpc error: code = NotFound desc = could not find container \"c13a844db1f9139f7a945c75d20d9997a276ce606e094cb1b22600e4ca62b2cf\": container with ID starting with c13a844db1f9139f7a945c75d20d9997a276ce606e094cb1b22600e4ca62b2cf not found: ID does not exist"
Sep 10 23:29:03.618591 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.618537    2563 scope.go:117] "RemoveContainer" containerID="e14fb6d58c1174d532c8028059a6b033d0f2c992ea0115f22556fbcea1ec0adf"
Sep 10 23:29:03.627415 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.627391    2563 scope.go:117] "RemoveContainer" containerID="e14fb6d58c1174d532c8028059a6b033d0f2c992ea0115f22556fbcea1ec0adf"
Sep 10 23:29:03.627772 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:29:03.627751    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e14fb6d58c1174d532c8028059a6b033d0f2c992ea0115f22556fbcea1ec0adf\": container with ID starting with e14fb6d58c1174d532c8028059a6b033d0f2c992ea0115f22556fbcea1ec0adf not found: ID does not exist" containerID="e14fb6d58c1174d532c8028059a6b033d0f2c992ea0115f22556fbcea1ec0adf"
Sep 10 23:29:03.627857 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.627781    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e14fb6d58c1174d532c8028059a6b033d0f2c992ea0115f22556fbcea1ec0adf"} err="failed to get container status \"e14fb6d58c1174d532c8028059a6b033d0f2c992ea0115f22556fbcea1ec0adf\": rpc error: code = NotFound desc = could not find container \"e14fb6d58c1174d532c8028059a6b033d0f2c992ea0115f22556fbcea1ec0adf\": container with ID starting with e14fb6d58c1174d532c8028059a6b033d0f2c992ea0115f22556fbcea1ec0adf not found: ID does not exist"
Sep 10 23:29:03.653184 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.653156    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5541bf40-af13-42a3-89d3-0c79e2a2b35b-plugins-config-volume\") pod \"5541bf40-af13-42a3-89d3-0c79e2a2b35b\" (UID: \"5541bf40-af13-42a3-89d3-0c79e2a2b35b\") "
Sep 10 23:29:03.653363 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.653218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q62dg\" (UniqueName: \"kubernetes.io/projected/5541bf40-af13-42a3-89d3-0c79e2a2b35b-kube-api-access-q62dg\") pod \"5541bf40-af13-42a3-89d3-0c79e2a2b35b\" (UID: \"5541bf40-af13-42a3-89d3-0c79e2a2b35b\") "
Sep 10 23:29:03.653363 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.653266    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg4ln\" (UniqueName: \"kubernetes.io/projected/f4084d5e-45e6-4acf-93d5-a9a9f2150b67-kube-api-access-tg4ln\") pod \"f4084d5e-45e6-4acf-93d5-a9a9f2150b67\" (UID: \"f4084d5e-45e6-4acf-93d5-a9a9f2150b67\") "
Sep 10 23:29:03.653363 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.653335    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4084d5e-45e6-4acf-93d5-a9a9f2150b67-plugins-config-volume\") pod \"f4084d5e-45e6-4acf-93d5-a9a9f2150b67\" (UID: \"f4084d5e-45e6-4acf-93d5-a9a9f2150b67\") "
Sep 10 23:29:03.653779 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.653743    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5541bf40-af13-42a3-89d3-0c79e2a2b35b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5541bf40-af13-42a3-89d3-0c79e2a2b35b" (UID: "5541bf40-af13-42a3-89d3-0c79e2a2b35b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:29:03.653889 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.653777    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4084d5e-45e6-4acf-93d5-a9a9f2150b67-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4084d5e-45e6-4acf-93d5-a9a9f2150b67" (UID: "f4084d5e-45e6-4acf-93d5-a9a9f2150b67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:29:03.655460 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.655428    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4084d5e-45e6-4acf-93d5-a9a9f2150b67-kube-api-access-tg4ln" (OuterVolumeSpecName: "kube-api-access-tg4ln") pod "f4084d5e-45e6-4acf-93d5-a9a9f2150b67" (UID: "f4084d5e-45e6-4acf-93d5-a9a9f2150b67"). InnerVolumeSpecName "kube-api-access-tg4ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:03.655460 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.655434    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5541bf40-af13-42a3-89d3-0c79e2a2b35b-kube-api-access-q62dg" (OuterVolumeSpecName: "kube-api-access-q62dg") pod "5541bf40-af13-42a3-89d3-0c79e2a2b35b" (UID: "5541bf40-af13-42a3-89d3-0c79e2a2b35b"). InnerVolumeSpecName "kube-api-access-q62dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:03.755305 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.755257    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q62dg\" (UniqueName: \"kubernetes.io/projected/5541bf40-af13-42a3-89d3-0c79e2a2b35b-kube-api-access-q62dg\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:03.755305 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.755300    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg4ln\" (UniqueName: \"kubernetes.io/projected/f4084d5e-45e6-4acf-93d5-a9a9f2150b67-kube-api-access-tg4ln\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:03.755305 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.755309    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4084d5e-45e6-4acf-93d5-a9a9f2150b67-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:03.755305 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.755317    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5541bf40-af13-42a3-89d3-0c79e2a2b35b-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:03.815277 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.815244    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b218306-51e7-4e3c-b2c0-416191435dec" path="/var/lib/kubelet/pods/7b218306-51e7-4e3c-b2c0-416191435dec/volumes"
Sep 10 23:29:03.815665 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.815651    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84598834-e1b5-490e-b44f-7dcdd2344ca9" path="/var/lib/kubelet/pods/84598834-e1b5-490e-b44f-7dcdd2344ca9/volumes"
Sep 10 23:29:03.923507 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.923460    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"]
Sep 10 23:29:03.926515 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.926480    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7025b9-6cd5b6ff89-9b7xv"]
Sep 10 23:29:03.938789 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.938751    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"]
Sep 10 23:29:03.945410 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:03.945385    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-7025b9-6cc7967d46-67pc7"]
Sep 10 23:29:05.813675 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:05.813639    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5541bf40-af13-42a3-89d3-0c79e2a2b35b" path="/var/lib/kubelet/pods/5541bf40-af13-42a3-89d3-0c79e2a2b35b/volumes"
Sep 10 23:29:05.814058 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:05.814006    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4084d5e-45e6-4acf-93d5-a9a9f2150b67" path="/var/lib/kubelet/pods/f4084d5e-45e6-4acf-93d5-a9a9f2150b67/volumes"
Sep 10 23:29:07.596081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.596011    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs" podUID="8dd338da-7bf3-45e5-85e1-94e9a6cbcd46" containerName="main" containerID="cri-o://e189549095db1e4f82fcb13f4522730ca64db5f3310c0ded3f95cfb78655b344" gracePeriod=45
Sep 10 23:29:07.598890 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.598854    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p" podUID="12269fd5-19ad-4396-9387-7e3c56ef50e7" containerName="main" containerID="cri-o://f9bff82d172da7b8999f6de38e295e2595de4ffe4833b1e58d73383ad6bc2750" gracePeriod=45
Sep 10 23:29:07.610102 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.610054    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2" podUID="548169f7-e180-41e8-bd45-2755dc4ed316" containerName="main" containerID="cri-o://1bbd27cb96c7e4ae0036ded507cf6d0b0422f9e9aef36c73cfda3f4558adf718" gracePeriod=45
Sep 10 23:29:07.616787 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.616756    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4" podUID="de1d1d9c-4d22-454d-8e6a-4d3c696af3d9" containerName="main" containerID="cri-o://bf1b4de9853c920d9a3652aa774340b043a0fa473310b5943cca6092fbce5ad4" gracePeriod=45
Sep 10 23:29:07.624646 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.624615    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq" podUID="637941ab-24a5-4220-a8ab-4f62c974654e" containerName="main" containerID="cri-o://9e523deb17d0d1a37cf4064c95139dbe6bc942f02e872193b61f21584e4804b1" gracePeriod=45
Sep 10 23:29:07.883080 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.883056    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:29:07.927253 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.927218    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:29:07.968410 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.968386    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:29:07.974682 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.974659    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:29:07.979261 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.979240    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:29:07.998093 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pffkc\" (UniqueName: \"kubernetes.io/projected/548169f7-e180-41e8-bd45-2755dc4ed316-kube-api-access-pffkc\") pod \"548169f7-e180-41e8-bd45-2755dc4ed316\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") "
Sep 10 23:29:07.998290 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998117    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-model-cache\") pod \"548169f7-e180-41e8-bd45-2755dc4ed316\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") "
Sep 10 23:29:07.998290 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998147    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-dshm\") pod \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") "
Sep 10 23:29:07.998290 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998261    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvdhz\" (UniqueName: \"kubernetes.io/projected/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-kube-api-access-tvdhz\") pod \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") "
Sep 10 23:29:07.998461 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998311    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/548169f7-e180-41e8-bd45-2755dc4ed316-tls-certs\") pod \"548169f7-e180-41e8-bd45-2755dc4ed316\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") "
Sep 10 23:29:07.998461 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998340    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-dshm\") pod \"548169f7-e180-41e8-bd45-2755dc4ed316\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") "
Sep 10 23:29:07.998461 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998376    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-tmp-dir\") pod \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") "
Sep 10 23:29:07.998461 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998391    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-model-cache" (OuterVolumeSpecName: "model-cache") pod "548169f7-e180-41e8-bd45-2755dc4ed316" (UID: "548169f7-e180-41e8-bd45-2755dc4ed316"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:07.998461 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998440    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-home\") pod \"548169f7-e180-41e8-bd45-2755dc4ed316\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") "
Sep 10 23:29:07.998767 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998481    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-model-cache\") pod \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") "
Sep 10 23:29:07.998767 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998550    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-home\") pod \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") "
Sep 10 23:29:07.998767 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998576    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-tls-certs\") pod \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\" (UID: \"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46\") "
Sep 10 23:29:07.998767 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998602    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-tmp-dir\") pod \"548169f7-e180-41e8-bd45-2755dc4ed316\" (UID: \"548169f7-e180-41e8-bd45-2755dc4ed316\") "
Sep 10 23:29:07.998767 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998739    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8dd338da-7bf3-45e5-85e1-94e9a6cbcd46" (UID: "8dd338da-7bf3-45e5-85e1-94e9a6cbcd46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:07.999044 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.998928    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-home" (OuterVolumeSpecName: "home") pod "8dd338da-7bf3-45e5-85e1-94e9a6cbcd46" (UID: "8dd338da-7bf3-45e5-85e1-94e9a6cbcd46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:07.999044 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.999001    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:07.999044 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.999017    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:07.999044 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.999029    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:07.999261 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.999100    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-home" (OuterVolumeSpecName: "home") pod "548169f7-e180-41e8-bd45-2755dc4ed316" (UID: "548169f7-e180-41e8-bd45-2755dc4ed316"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:07.999320 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.999291    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-model-cache" (OuterVolumeSpecName: "model-cache") pod "8dd338da-7bf3-45e5-85e1-94e9a6cbcd46" (UID: "8dd338da-7bf3-45e5-85e1-94e9a6cbcd46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:07.999514 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:07.999473    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "548169f7-e180-41e8-bd45-2755dc4ed316" (UID: "548169f7-e180-41e8-bd45-2755dc4ed316"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.001522 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.001477    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548169f7-e180-41e8-bd45-2755dc4ed316-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "548169f7-e180-41e8-bd45-2755dc4ed316" (UID: "548169f7-e180-41e8-bd45-2755dc4ed316"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:29:08.001801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.001780    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-dshm" (OuterVolumeSpecName: "dshm") pod "8dd338da-7bf3-45e5-85e1-94e9a6cbcd46" (UID: "8dd338da-7bf3-45e5-85e1-94e9a6cbcd46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.001990 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.001968    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8dd338da-7bf3-45e5-85e1-94e9a6cbcd46" (UID: "8dd338da-7bf3-45e5-85e1-94e9a6cbcd46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:29:08.002271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.002153    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548169f7-e180-41e8-bd45-2755dc4ed316-kube-api-access-pffkc" (OuterVolumeSpecName: "kube-api-access-pffkc") pod "548169f7-e180-41e8-bd45-2755dc4ed316" (UID: "548169f7-e180-41e8-bd45-2755dc4ed316"). InnerVolumeSpecName "kube-api-access-pffkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:08.002624 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.002530    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-dshm" (OuterVolumeSpecName: "dshm") pod "548169f7-e180-41e8-bd45-2755dc4ed316" (UID: "548169f7-e180-41e8-bd45-2755dc4ed316"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.003849 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.003823    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-kube-api-access-tvdhz" (OuterVolumeSpecName: "kube-api-access-tvdhz") pod "8dd338da-7bf3-45e5-85e1-94e9a6cbcd46" (UID: "8dd338da-7bf3-45e5-85e1-94e9a6cbcd46"). InnerVolumeSpecName "kube-api-access-tvdhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:08.100151 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100119    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x8lb\" (UniqueName: \"kubernetes.io/projected/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-kube-api-access-4x8lb\") pod \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") "
Sep 10 23:29:08.100334 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100166    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-model-cache\") pod \"637941ab-24a5-4220-a8ab-4f62c974654e\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") "
Sep 10 23:29:08.100334 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100206    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-home\") pod \"12269fd5-19ad-4396-9387-7e3c56ef50e7\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") "
Sep 10 23:29:08.100334 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100236    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-dshm\") pod \"12269fd5-19ad-4396-9387-7e3c56ef50e7\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") "
Sep 10 23:29:08.100334 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100271    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-home\") pod \"637941ab-24a5-4220-a8ab-4f62c974654e\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") "
Sep 10 23:29:08.100334 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100299    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637941ab-24a5-4220-a8ab-4f62c974654e-tls-certs\") pod \"637941ab-24a5-4220-a8ab-4f62c974654e\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") "
Sep 10 23:29:08.100669 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100337    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdstr\" (UniqueName: \"kubernetes.io/projected/12269fd5-19ad-4396-9387-7e3c56ef50e7-kube-api-access-wdstr\") pod \"12269fd5-19ad-4396-9387-7e3c56ef50e7\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") "
Sep 10 23:29:08.100669 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100379    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12269fd5-19ad-4396-9387-7e3c56ef50e7-tls-certs\") pod \"12269fd5-19ad-4396-9387-7e3c56ef50e7\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") "
Sep 10 23:29:08.100669 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100417    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hghqq\" (UniqueName: \"kubernetes.io/projected/637941ab-24a5-4220-a8ab-4f62c974654e-kube-api-access-hghqq\") pod \"637941ab-24a5-4220-a8ab-4f62c974654e\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") "
Sep 10 23:29:08.100669 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100458    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-tmp-dir\") pod \"637941ab-24a5-4220-a8ab-4f62c974654e\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") "
Sep 10 23:29:08.100669 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100487    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-tmp-dir\") pod \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") "
Sep 10 23:29:08.100669 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100525    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-model-cache" (OuterVolumeSpecName: "model-cache") pod "637941ab-24a5-4220-a8ab-4f62c974654e" (UID: "637941ab-24a5-4220-a8ab-4f62c974654e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.100669 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100542    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-dshm\") pod \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") "
Sep 10 23:29:08.100669 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100643    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-model-cache\") pod \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") "
Sep 10 23:29:08.101096 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100680    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-home" (OuterVolumeSpecName: "home") pod "12269fd5-19ad-4396-9387-7e3c56ef50e7" (UID: "12269fd5-19ad-4396-9387-7e3c56ef50e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.101096 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100699    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-dshm\") pod \"637941ab-24a5-4220-a8ab-4f62c974654e\" (UID: \"637941ab-24a5-4220-a8ab-4f62c974654e\") "
Sep 10 23:29:08.101096 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100724    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-model-cache\") pod \"12269fd5-19ad-4396-9387-7e3c56ef50e7\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") "
Sep 10 23:29:08.101096 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100747    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-tmp-dir\") pod \"12269fd5-19ad-4396-9387-7e3c56ef50e7\" (UID: \"12269fd5-19ad-4396-9387-7e3c56ef50e7\") "
Sep 10 23:29:08.101096 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100776    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-home\") pod \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") "
Sep 10 23:29:08.101096 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100807    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-tls-certs\") pod \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\" (UID: \"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9\") "
Sep 10 23:29:08.101096 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.100897    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-home" (OuterVolumeSpecName: "home") pod "637941ab-24a5-4220-a8ab-4f62c974654e" (UID: "637941ab-24a5-4220-a8ab-4f62c974654e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.101484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101187    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pffkc\" (UniqueName: \"kubernetes.io/projected/548169f7-e180-41e8-bd45-2755dc4ed316-kube-api-access-pffkc\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.101484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101207    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.101484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101221    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.101484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101237    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvdhz\" (UniqueName: \"kubernetes.io/projected/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-kube-api-access-tvdhz\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.101484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101251    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/548169f7-e180-41e8-bd45-2755dc4ed316-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.101484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101264    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.101484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101276    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.101484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101286    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.101484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101297    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.101484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101310    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.101484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101321    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.101484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101333    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/548169f7-e180-41e8-bd45-2755dc4ed316-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.102103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101511    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "12269fd5-19ad-4396-9387-7e3c56ef50e7" (UID: "12269fd5-19ad-4396-9387-7e3c56ef50e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.102103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101705    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-home" (OuterVolumeSpecName: "home") pod "de1d1d9c-4d22-454d-8e6a-4d3c696af3d9" (UID: "de1d1d9c-4d22-454d-8e6a-4d3c696af3d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.102103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101710    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12269fd5-19ad-4396-9387-7e3c56ef50e7" (UID: "12269fd5-19ad-4396-9387-7e3c56ef50e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.102103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.101717    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "de1d1d9c-4d22-454d-8e6a-4d3c696af3d9" (UID: "de1d1d9c-4d22-454d-8e6a-4d3c696af3d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.102873 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.102664    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "637941ab-24a5-4220-a8ab-4f62c974654e" (UID: "637941ab-24a5-4220-a8ab-4f62c974654e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.102873 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.102846    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de1d1d9c-4d22-454d-8e6a-4d3c696af3d9" (UID: "de1d1d9c-4d22-454d-8e6a-4d3c696af3d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.103054 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.102968    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-dshm" (OuterVolumeSpecName: "dshm") pod "12269fd5-19ad-4396-9387-7e3c56ef50e7" (UID: "12269fd5-19ad-4396-9387-7e3c56ef50e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.103791 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.103666    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12269fd5-19ad-4396-9387-7e3c56ef50e7-kube-api-access-wdstr" (OuterVolumeSpecName: "kube-api-access-wdstr") pod "12269fd5-19ad-4396-9387-7e3c56ef50e7" (UID: "12269fd5-19ad-4396-9387-7e3c56ef50e7"). InnerVolumeSpecName "kube-api-access-wdstr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:08.103791 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.103747    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-dshm" (OuterVolumeSpecName: "dshm") pod "de1d1d9c-4d22-454d-8e6a-4d3c696af3d9" (UID: "de1d1d9c-4d22-454d-8e6a-4d3c696af3d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.104080 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.104057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de1d1d9c-4d22-454d-8e6a-4d3c696af3d9" (UID: "de1d1d9c-4d22-454d-8e6a-4d3c696af3d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:29:08.104181 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.104075    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/637941ab-24a5-4220-a8ab-4f62c974654e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "637941ab-24a5-4220-a8ab-4f62c974654e" (UID: "637941ab-24a5-4220-a8ab-4f62c974654e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:29:08.104181 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.104099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-dshm" (OuterVolumeSpecName: "dshm") pod "637941ab-24a5-4220-a8ab-4f62c974654e" (UID: "637941ab-24a5-4220-a8ab-4f62c974654e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:29:08.104615 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.104594    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-kube-api-access-4x8lb" (OuterVolumeSpecName: "kube-api-access-4x8lb") pod "de1d1d9c-4d22-454d-8e6a-4d3c696af3d9" (UID: "de1d1d9c-4d22-454d-8e6a-4d3c696af3d9"). InnerVolumeSpecName "kube-api-access-4x8lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:08.104730 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.104715    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12269fd5-19ad-4396-9387-7e3c56ef50e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12269fd5-19ad-4396-9387-7e3c56ef50e7" (UID: "12269fd5-19ad-4396-9387-7e3c56ef50e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:29:08.105358 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.105333    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637941ab-24a5-4220-a8ab-4f62c974654e-kube-api-access-hghqq" (OuterVolumeSpecName: "kube-api-access-hghqq") pod "637941ab-24a5-4220-a8ab-4f62c974654e" (UID: "637941ab-24a5-4220-a8ab-4f62c974654e"). InnerVolumeSpecName "kube-api-access-hghqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:08.202527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202410    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.202527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202442    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.202527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202452    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.202527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202460    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.202527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202468    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.202527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202475    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.202527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202484    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x8lb\" (UniqueName: \"kubernetes.io/projected/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-kube-api-access-4x8lb\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.202527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202521    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12269fd5-19ad-4396-9387-7e3c56ef50e7-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.202527 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202534    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/637941ab-24a5-4220-a8ab-4f62c974654e-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.202954 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202547    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdstr\" (UniqueName: \"kubernetes.io/projected/12269fd5-19ad-4396-9387-7e3c56ef50e7-kube-api-access-wdstr\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.202954 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202561    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12269fd5-19ad-4396-9387-7e3c56ef50e7-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.202954 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202573    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hghqq\" (UniqueName: \"kubernetes.io/projected/637941ab-24a5-4220-a8ab-4f62c974654e-kube-api-access-hghqq\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.202954 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202583    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/637941ab-24a5-4220-a8ab-4f62c974654e-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.202954 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202592    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.202954 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.202599    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:29:08.638989 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.638951    2563 generic.go:356] "Generic (PLEG): container finished" podID="637941ab-24a5-4220-a8ab-4f62c974654e" containerID="9e523deb17d0d1a37cf4064c95139dbe6bc942f02e872193b61f21584e4804b1" exitCode=0
Sep 10 23:29:08.639406 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.639007    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"
Sep 10 23:29:08.639406 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.639043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq" event={"ID":"637941ab-24a5-4220-a8ab-4f62c974654e","Type":"ContainerDied","Data":"9e523deb17d0d1a37cf4064c95139dbe6bc942f02e872193b61f21584e4804b1"}
Sep 10 23:29:08.639406 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.639103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq" event={"ID":"637941ab-24a5-4220-a8ab-4f62c974654e","Type":"ContainerDied","Data":"71e61490a1d4a6bdc7ee0101d64c4509475343eee7a508344c6fd81e17133312"}
Sep 10 23:29:08.639406 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.639130    2563 scope.go:117] "RemoveContainer" containerID="9e523deb17d0d1a37cf4064c95139dbe6bc942f02e872193b61f21584e4804b1"
Sep 10 23:29:08.640461 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.640434    2563 generic.go:356] "Generic (PLEG): container finished" podID="548169f7-e180-41e8-bd45-2755dc4ed316" containerID="1bbd27cb96c7e4ae0036ded507cf6d0b0422f9e9aef36c73cfda3f4558adf718" exitCode=0
Sep 10 23:29:08.640552 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.640484    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"
Sep 10 23:29:08.640552 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.640528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2" event={"ID":"548169f7-e180-41e8-bd45-2755dc4ed316","Type":"ContainerDied","Data":"1bbd27cb96c7e4ae0036ded507cf6d0b0422f9e9aef36c73cfda3f4558adf718"}
Sep 10 23:29:08.640916 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.640866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2" event={"ID":"548169f7-e180-41e8-bd45-2755dc4ed316","Type":"ContainerDied","Data":"f685d624b840013e9f5fcdb0e8222c56c676d6e9beb3d4bc18eb575d0851731f"}
Sep 10 23:29:08.642231 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.642205    2563 generic.go:356] "Generic (PLEG): container finished" podID="de1d1d9c-4d22-454d-8e6a-4d3c696af3d9" containerID="bf1b4de9853c920d9a3652aa774340b043a0fa473310b5943cca6092fbce5ad4" exitCode=0
Sep 10 23:29:08.642352 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.642275    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"
Sep 10 23:29:08.642352 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.642286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4" event={"ID":"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9","Type":"ContainerDied","Data":"bf1b4de9853c920d9a3652aa774340b043a0fa473310b5943cca6092fbce5ad4"}
Sep 10 23:29:08.642352 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.642326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4" event={"ID":"de1d1d9c-4d22-454d-8e6a-4d3c696af3d9","Type":"ContainerDied","Data":"84cb7b028823dad9480d28fc8652f535d9f689f9a349c64f118f0a3ef7a8d12c"}
Sep 10 23:29:08.643885 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.643864    2563 generic.go:356] "Generic (PLEG): container finished" podID="8dd338da-7bf3-45e5-85e1-94e9a6cbcd46" containerID="e189549095db1e4f82fcb13f4522730ca64db5f3310c0ded3f95cfb78655b344" exitCode=0
Sep 10 23:29:08.644037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.643930    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"
Sep 10 23:29:08.644037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.643950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs" event={"ID":"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46","Type":"ContainerDied","Data":"e189549095db1e4f82fcb13f4522730ca64db5f3310c0ded3f95cfb78655b344"}
Sep 10 23:29:08.644037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.643979    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs" event={"ID":"8dd338da-7bf3-45e5-85e1-94e9a6cbcd46","Type":"ContainerDied","Data":"e119ac3966a65fa00564605f42d21936fa28465b3fded8d0615846a77a1bae9e"}
Sep 10 23:29:08.645844 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.645820    2563 generic.go:356] "Generic (PLEG): container finished" podID="12269fd5-19ad-4396-9387-7e3c56ef50e7" containerID="f9bff82d172da7b8999f6de38e295e2595de4ffe4833b1e58d73383ad6bc2750" exitCode=0
Sep 10 23:29:08.645942 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.645868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p" event={"ID":"12269fd5-19ad-4396-9387-7e3c56ef50e7","Type":"ContainerDied","Data":"f9bff82d172da7b8999f6de38e295e2595de4ffe4833b1e58d73383ad6bc2750"}
Sep 10 23:29:08.645942 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.645874    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"
Sep 10 23:29:08.645942 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.645891    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p" event={"ID":"12269fd5-19ad-4396-9387-7e3c56ef50e7","Type":"ContainerDied","Data":"8e5e1d3b6e32e99d24c96669ccf4dfface6c3580cf91d374910b4edf250f2df4"}
Sep 10 23:29:08.649861 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.649842    2563 scope.go:117] "RemoveContainer" containerID="9e523deb17d0d1a37cf4064c95139dbe6bc942f02e872193b61f21584e4804b1"
Sep 10 23:29:08.650690 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:29:08.650664    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e523deb17d0d1a37cf4064c95139dbe6bc942f02e872193b61f21584e4804b1\": container with ID starting with 9e523deb17d0d1a37cf4064c95139dbe6bc942f02e872193b61f21584e4804b1 not found: ID does not exist" containerID="9e523deb17d0d1a37cf4064c95139dbe6bc942f02e872193b61f21584e4804b1"
Sep 10 23:29:08.650769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.650702    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e523deb17d0d1a37cf4064c95139dbe6bc942f02e872193b61f21584e4804b1"} err="failed to get container status \"9e523deb17d0d1a37cf4064c95139dbe6bc942f02e872193b61f21584e4804b1\": rpc error: code = NotFound desc = could not find container \"9e523deb17d0d1a37cf4064c95139dbe6bc942f02e872193b61f21584e4804b1\": container with ID starting with 9e523deb17d0d1a37cf4064c95139dbe6bc942f02e872193b61f21584e4804b1 not found: ID does not exist"
Sep 10 23:29:08.650769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.650725    2563 scope.go:117] "RemoveContainer" containerID="1bbd27cb96c7e4ae0036ded507cf6d0b0422f9e9aef36c73cfda3f4558adf718"
Sep 10 23:29:08.660992 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.660974    2563 scope.go:117] "RemoveContainer" containerID="1bbd27cb96c7e4ae0036ded507cf6d0b0422f9e9aef36c73cfda3f4558adf718"
Sep 10 23:29:08.661245 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:29:08.661230    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bbd27cb96c7e4ae0036ded507cf6d0b0422f9e9aef36c73cfda3f4558adf718\": container with ID starting with 1bbd27cb96c7e4ae0036ded507cf6d0b0422f9e9aef36c73cfda3f4558adf718 not found: ID does not exist" containerID="1bbd27cb96c7e4ae0036ded507cf6d0b0422f9e9aef36c73cfda3f4558adf718"
Sep 10 23:29:08.661291 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.661251    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbd27cb96c7e4ae0036ded507cf6d0b0422f9e9aef36c73cfda3f4558adf718"} err="failed to get container status \"1bbd27cb96c7e4ae0036ded507cf6d0b0422f9e9aef36c73cfda3f4558adf718\": rpc error: code = NotFound desc = could not find container \"1bbd27cb96c7e4ae0036ded507cf6d0b0422f9e9aef36c73cfda3f4558adf718\": container with ID starting with 1bbd27cb96c7e4ae0036ded507cf6d0b0422f9e9aef36c73cfda3f4558adf718 not found: ID does not exist"
Sep 10 23:29:08.661291 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.661266    2563 scope.go:117] "RemoveContainer" containerID="bf1b4de9853c920d9a3652aa774340b043a0fa473310b5943cca6092fbce5ad4"
Sep 10 23:29:08.666909 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.666885    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"]
Sep 10 23:29:08.668763 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.668741    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/simulator-w4-7025b9-kserve-c98c9d948-nndrq"]
Sep 10 23:29:08.670188 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.670170    2563 scope.go:117] "RemoveContainer" containerID="bf1b4de9853c920d9a3652aa774340b043a0fa473310b5943cca6092fbce5ad4"
Sep 10 23:29:08.670435 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:29:08.670417    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1b4de9853c920d9a3652aa774340b043a0fa473310b5943cca6092fbce5ad4\": container with ID starting with bf1b4de9853c920d9a3652aa774340b043a0fa473310b5943cca6092fbce5ad4 not found: ID does not exist" containerID="bf1b4de9853c920d9a3652aa774340b043a0fa473310b5943cca6092fbce5ad4"
Sep 10 23:29:08.670483 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.670443    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1b4de9853c920d9a3652aa774340b043a0fa473310b5943cca6092fbce5ad4"} err="failed to get container status \"bf1b4de9853c920d9a3652aa774340b043a0fa473310b5943cca6092fbce5ad4\": rpc error: code = NotFound desc = could not find container \"bf1b4de9853c920d9a3652aa774340b043a0fa473310b5943cca6092fbce5ad4\": container with ID starting with bf1b4de9853c920d9a3652aa774340b043a0fa473310b5943cca6092fbce5ad4 not found: ID does not exist"
Sep 10 23:29:08.670483 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.670459    2563 scope.go:117] "RemoveContainer" containerID="e189549095db1e4f82fcb13f4522730ca64db5f3310c0ded3f95cfb78655b344"
Sep 10 23:29:08.677411 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.677387    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"]
Sep 10 23:29:08.679433 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.679413    2563 scope.go:117] "RemoveContainer" containerID="e189549095db1e4f82fcb13f4522730ca64db5f3310c0ded3f95cfb78655b344"
Sep 10 23:29:08.679782 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:29:08.679763    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e189549095db1e4f82fcb13f4522730ca64db5f3310c0ded3f95cfb78655b344\": container with ID starting with e189549095db1e4f82fcb13f4522730ca64db5f3310c0ded3f95cfb78655b344 not found: ID does not exist" containerID="e189549095db1e4f82fcb13f4522730ca64db5f3310c0ded3f95cfb78655b344"
Sep 10 23:29:08.679849 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.679790    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e189549095db1e4f82fcb13f4522730ca64db5f3310c0ded3f95cfb78655b344"} err="failed to get container status \"e189549095db1e4f82fcb13f4522730ca64db5f3310c0ded3f95cfb78655b344\": rpc error: code = NotFound desc = could not find container \"e189549095db1e4f82fcb13f4522730ca64db5f3310c0ded3f95cfb78655b344\": container with ID starting with e189549095db1e4f82fcb13f4522730ca64db5f3310c0ded3f95cfb78655b344 not found: ID does not exist"
Sep 10 23:29:08.679849 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.679809    2563 scope.go:117] "RemoveContainer" containerID="f9bff82d172da7b8999f6de38e295e2595de4ffe4833b1e58d73383ad6bc2750"
Sep 10 23:29:08.680199 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.680181    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/distinct-w4-7025b9-kserve-68585676f6-k2w6p"]
Sep 10 23:29:08.688107 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.688087    2563 scope.go:117] "RemoveContainer" containerID="f9bff82d172da7b8999f6de38e295e2595de4ffe4833b1e58d73383ad6bc2750"
Sep 10 23:29:08.688472 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:29:08.688449    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9bff82d172da7b8999f6de38e295e2595de4ffe4833b1e58d73383ad6bc2750\": container with ID starting with f9bff82d172da7b8999f6de38e295e2595de4ffe4833b1e58d73383ad6bc2750 not found: ID does not exist" containerID="f9bff82d172da7b8999f6de38e295e2595de4ffe4833b1e58d73383ad6bc2750"
Sep 10 23:29:08.688577 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.688475    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9bff82d172da7b8999f6de38e295e2595de4ffe4833b1e58d73383ad6bc2750"} err="failed to get container status \"f9bff82d172da7b8999f6de38e295e2595de4ffe4833b1e58d73383ad6bc2750\": rpc error: code = NotFound desc = could not find container \"f9bff82d172da7b8999f6de38e295e2595de4ffe4833b1e58d73383ad6bc2750\": container with ID starting with f9bff82d172da7b8999f6de38e295e2595de4ffe4833b1e58d73383ad6bc2750 not found: ID does not exist"
Sep 10 23:29:08.690174 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.690151    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"]
Sep 10 23:29:08.691933 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.691910    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/embedding-w4-7025b9-kserve-7544fb77c-f24g2"]
Sep 10 23:29:08.702591 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.702566    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"]
Sep 10 23:29:08.704222 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.704197    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/premium-w4-7025b9-kserve-7f98995769-sbbf4"]
Sep 10 23:29:08.712992 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.712972    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"]
Sep 10 23:29:08.718932 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:08.718911    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7025b9/distinct-2-w4-7025b9-kserve-778b6c9955-wk4fs"]
Sep 10 23:29:09.815023 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:09.814990    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12269fd5-19ad-4396-9387-7e3c56ef50e7" path="/var/lib/kubelet/pods/12269fd5-19ad-4396-9387-7e3c56ef50e7/volumes"
Sep 10 23:29:09.815392 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:09.815374    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="548169f7-e180-41e8-bd45-2755dc4ed316" path="/var/lib/kubelet/pods/548169f7-e180-41e8-bd45-2755dc4ed316/volumes"
Sep 10 23:29:09.815748 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:09.815734    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="637941ab-24a5-4220-a8ab-4f62c974654e" path="/var/lib/kubelet/pods/637941ab-24a5-4220-a8ab-4f62c974654e/volumes"
Sep 10 23:29:09.816367 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:09.816351    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dd338da-7bf3-45e5-85e1-94e9a6cbcd46" path="/var/lib/kubelet/pods/8dd338da-7bf3-45e5-85e1-94e9a6cbcd46/volumes"
Sep 10 23:29:09.816739 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:09.816727    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de1d1d9c-4d22-454d-8e6a-4d3c696af3d9" path="/var/lib/kubelet/pods/de1d1d9c-4d22-454d-8e6a-4d3c696af3d9/volumes"
Sep 10 23:29:20.204593 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:20.204562    2563 scope.go:117] "RemoveContainer" containerID="11d17288e6f509420fcff9736145d4dc69ce518646b649999ab60f66de16d5da"
Sep 10 23:29:35.334174 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334141    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"]
Sep 10 23:29:35.334756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334697    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548169f7-e180-41e8-bd45-2755dc4ed316" containerName="main"
Sep 10 23:29:35.334756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334715    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="548169f7-e180-41e8-bd45-2755dc4ed316" containerName="main"
Sep 10 23:29:35.334756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334730    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5541bf40-af13-42a3-89d3-0c79e2a2b35b" containerName="payload-processing"
Sep 10 23:29:35.334756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334739    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5541bf40-af13-42a3-89d3-0c79e2a2b35b" containerName="payload-processing"
Sep 10 23:29:35.334756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334750    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de1d1d9c-4d22-454d-8e6a-4d3c696af3d9" containerName="main"
Sep 10 23:29:35.334756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334759    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="de1d1d9c-4d22-454d-8e6a-4d3c696af3d9" containerName="main"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334775    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="637941ab-24a5-4220-a8ab-4f62c974654e" containerName="main"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334780    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="637941ab-24a5-4220-a8ab-4f62c974654e" containerName="main"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334785    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12269fd5-19ad-4396-9387-7e3c56ef50e7" containerName="main"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334790    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="12269fd5-19ad-4396-9387-7e3c56ef50e7" containerName="main"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334797    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dd338da-7bf3-45e5-85e1-94e9a6cbcd46" containerName="main"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334803    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dd338da-7bf3-45e5-85e1-94e9a6cbcd46" containerName="main"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334815    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84598834-e1b5-490e-b44f-7dcdd2344ca9" containerName="revoke-keys"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334824    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="84598834-e1b5-490e-b44f-7dcdd2344ca9" containerName="revoke-keys"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334838    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4084d5e-45e6-4acf-93d5-a9a9f2150b67" containerName="payload-pre-processing"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334843    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4084d5e-45e6-4acf-93d5-a9a9f2150b67" containerName="payload-pre-processing"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334904    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4084d5e-45e6-4acf-93d5-a9a9f2150b67" containerName="payload-pre-processing"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334912    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="548169f7-e180-41e8-bd45-2755dc4ed316" containerName="main"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334919    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="de1d1d9c-4d22-454d-8e6a-4d3c696af3d9" containerName="main"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334927    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="84598834-e1b5-490e-b44f-7dcdd2344ca9" containerName="revoke-keys"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334932    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="637941ab-24a5-4220-a8ab-4f62c974654e" containerName="main"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334938    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="12269fd5-19ad-4396-9387-7e3c56ef50e7" containerName="main"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334942    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dd338da-7bf3-45e5-85e1-94e9a6cbcd46" containerName="main"
Sep 10 23:29:35.335099 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.334948    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5541bf40-af13-42a3-89d3-0c79e2a2b35b" containerName="payload-processing"
Sep 10 23:29:35.338158 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.338138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.340852 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.340830    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-83be66\"/\"sub-test-model-83be66-kserve-self-signed-certs\""
Sep 10 23:29:35.340978 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.340907    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-83be66\"/\"openshift-service-ca.crt\""
Sep 10 23:29:35.340978 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.340942    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-83be66\"/\"kube-root-ca.crt\""
Sep 10 23:29:35.340978 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.340951    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-83be66\"/\"default-dockercfg-dq6xc\""
Sep 10 23:29:35.349186 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.349165    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"]
Sep 10 23:29:35.357300 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.357276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de5e873-9b5a-4184-aa35-f250ca59369b-tls-certs\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.357409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.357326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-home\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.357409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.357361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-tmp-dir\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.357409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.357395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6p6t\" (UniqueName: \"kubernetes.io/projected/6de5e873-9b5a-4184-aa35-f250ca59369b-kube-api-access-s6p6t\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.357565 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.357469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-dshm\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.357626 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.357562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-model-cache\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.458559 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.458519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-model-cache\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.458720 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.458639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de5e873-9b5a-4184-aa35-f250ca59369b-tls-certs\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.458720 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.458672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-home\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.458720 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.458698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-tmp-dir\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.458896 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.458728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6p6t\" (UniqueName: \"kubernetes.io/projected/6de5e873-9b5a-4184-aa35-f250ca59369b-kube-api-access-s6p6t\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.458896 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.458765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-dshm\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.459005 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.458898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-model-cache\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.459005 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.458962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-home\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.459116 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.459014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-tmp-dir\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.460897 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.460875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-dshm\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.461118 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.461104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de5e873-9b5a-4184-aa35-f250ca59369b-tls-certs\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.465871 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.465849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6p6t\" (UniqueName: \"kubernetes.io/projected/6de5e873-9b5a-4184-aa35-f250ca59369b-kube-api-access-s6p6t\") pod \"sub-test-model-83be66-kserve-d8fdb4d9f-875fs\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") " pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.649807 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.649727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:35.920571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:35.920547    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"]
Sep 10 23:29:35.922555 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:29:35.922526    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6de5e873_9b5a_4184_aa35_f250ca59369b.slice/crio-82e68691345a2f2488e528d6773048ef1b574204adf26020ebfe103091e94cf9 WatchSource:0}: Error finding container 82e68691345a2f2488e528d6773048ef1b574204adf26020ebfe103091e94cf9: Status 404 returned error can't find the container with id 82e68691345a2f2488e528d6773048ef1b574204adf26020ebfe103091e94cf9
Sep 10 23:29:36.798831 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:36.798792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs" event={"ID":"6de5e873-9b5a-4184-aa35-f250ca59369b","Type":"ContainerStarted","Data":"5229ba5fba95e3e21cb3e278cf9f4d67554300bc08fa9ed8adc095b0caf17574"}
Sep 10 23:29:36.799202 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:36.798840    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs" event={"ID":"6de5e873-9b5a-4184-aa35-f250ca59369b","Type":"ContainerStarted","Data":"82e68691345a2f2488e528d6773048ef1b574204adf26020ebfe103091e94cf9"}
Sep 10 23:29:36.816112 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:36.816061    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs" podStartSLOduration=1.8160470260000001 podStartE2EDuration="1.816047026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:29:36.814577471 +0000 UTC m=+1697.506154656" watchObservedRunningTime="2026-09-10 23:29:36.816047026 +0000 UTC m=+1697.507624209"
Sep 10 23:29:45.650205 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:45.650111    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:45.650205 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:45.650166    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:45.662700 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:45.662677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:29:45.854319 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:29:45.854280    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:30:00.133692 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.133659    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-83be66-29818050-w8sdd"]
Sep 10 23:30:00.137200 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.137184    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818050-h78xl"]
Sep 10 23:30:00.137315 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.137296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-83be66-29818050-w8sdd"
Sep 10 23:30:00.139544 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.139521    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:30:00.141517 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.141486    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818050-h78xl"
Sep 10 23:30:00.144633 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.144613    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818050-h78xl"]
Sep 10 23:30:00.148416 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.148396    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-83be66-29818050-w8sdd"]
Sep 10 23:30:00.152756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.152736    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-83be66-29818050-w8sdd"
Sep 10 23:30:00.160007 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.159985    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818050-h78xl"
Sep 10 23:30:00.429728 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.429701    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-83be66-29818050-w8sdd"]
Sep 10 23:30:00.431393 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:30:00.431370    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8819aba5_4867_4d59_bad2_b74f8c9c2ed2.slice/crio-d8b5cdd13a76fc48df3c43194c32a810d051c76511dbbefed124458bcd7d1e12 WatchSource:0}: Error finding container d8b5cdd13a76fc48df3c43194c32a810d051c76511dbbefed124458bcd7d1e12: Status 404 returned error can't find the container with id d8b5cdd13a76fc48df3c43194c32a810d051c76511dbbefed124458bcd7d1e12
Sep 10 23:30:00.439439 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.439418    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818050-h78xl"]
Sep 10 23:30:00.444835 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:30:00.444809    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda643cba3_f7d7_4070_a8d9_df6cf2f54445.slice/crio-4ac4d79b5cc12362da1a2a8d200e0c3b41c637375e5b5b9910bf9ab6e32f9319 WatchSource:0}: Error finding container 4ac4d79b5cc12362da1a2a8d200e0c3b41c637375e5b5b9910bf9ab6e32f9319: Status 404 returned error can't find the container with id 4ac4d79b5cc12362da1a2a8d200e0c3b41c637375e5b5b9910bf9ab6e32f9319
Sep 10 23:30:00.919334 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.919295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818050-h78xl" event={"ID":"a643cba3-f7d7-4070-a8d9-df6cf2f54445","Type":"ContainerStarted","Data":"5b5c22399d7c6cde666f99ac7e1b41ac3a9d22ce1e46117e953e7de6060cc322"}
Sep 10 23:30:00.919547 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.919346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818050-h78xl" event={"ID":"a643cba3-f7d7-4070-a8d9-df6cf2f54445","Type":"ContainerStarted","Data":"4ac4d79b5cc12362da1a2a8d200e0c3b41c637375e5b5b9910bf9ab6e32f9319"}
Sep 10 23:30:00.920836 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.920814    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-83be66-29818050-w8sdd" event={"ID":"8819aba5-4867-4d59-bad2-b74f8c9c2ed2","Type":"ContainerStarted","Data":"3ec911f636e279833fb32029abef3abb3ec0c201f5a2d03974627eeea05e1880"}
Sep 10 23:30:00.920938 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.920855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-83be66-29818050-w8sdd" event={"ID":"8819aba5-4867-4d59-bad2-b74f8c9c2ed2","Type":"ContainerStarted","Data":"d8b5cdd13a76fc48df3c43194c32a810d051c76511dbbefed124458bcd7d1e12"}
Sep 10 23:30:00.933756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.933712    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818050-h78xl" podStartSLOduration=0.933697777 podStartE2EDuration="933.697777ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:30:00.932663468 +0000 UTC m=+1721.624240677" watchObservedRunningTime="2026-09-10 23:30:00.933697777 +0000 UTC m=+1721.625274961"
Sep 10 23:30:00.947409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:00.947356    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-83be66-29818050-w8sdd" podStartSLOduration=0.94734177 podStartE2EDuration="947.34177ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:30:00.945376854 +0000 UTC m=+1721.636954057" watchObservedRunningTime="2026-09-10 23:30:00.94734177 +0000 UTC m=+1721.638918954"
Sep 10 23:30:01.928089 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:01.928050    2563 generic.go:356] "Generic (PLEG): container finished" podID="a643cba3-f7d7-4070-a8d9-df6cf2f54445" containerID="5b5c22399d7c6cde666f99ac7e1b41ac3a9d22ce1e46117e953e7de6060cc322" exitCode=0
Sep 10 23:30:01.928587 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:01.928134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818050-h78xl" event={"ID":"a643cba3-f7d7-4070-a8d9-df6cf2f54445","Type":"ContainerDied","Data":"5b5c22399d7c6cde666f99ac7e1b41ac3a9d22ce1e46117e953e7de6060cc322"}
Sep 10 23:30:01.929743 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:01.929724    2563 generic.go:356] "Generic (PLEG): container finished" podID="8819aba5-4867-4d59-bad2-b74f8c9c2ed2" containerID="3ec911f636e279833fb32029abef3abb3ec0c201f5a2d03974627eeea05e1880" exitCode=0
Sep 10 23:30:01.929826 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:01.929751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-83be66-29818050-w8sdd" event={"ID":"8819aba5-4867-4d59-bad2-b74f8c9c2ed2","Type":"ContainerDied","Data":"3ec911f636e279833fb32029abef3abb3ec0c201f5a2d03974627eeea05e1880"}
Sep 10 23:30:03.058416 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:03.058394    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-83be66-29818050-w8sdd"
Sep 10 23:30:03.096145 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:03.096115    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818050-h78xl"
Sep 10 23:30:03.941937 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:03.941903    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-83be66-29818050-w8sdd"
Sep 10 23:30:03.942092 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:03.941926    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-83be66-29818050-w8sdd" event={"ID":"8819aba5-4867-4d59-bad2-b74f8c9c2ed2","Type":"ContainerDied","Data":"d8b5cdd13a76fc48df3c43194c32a810d051c76511dbbefed124458bcd7d1e12"}
Sep 10 23:30:03.942092 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:03.941983    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8b5cdd13a76fc48df3c43194c32a810d051c76511dbbefed124458bcd7d1e12"
Sep 10 23:30:03.943261 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:03.943243    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818050-h78xl"
Sep 10 23:30:03.943261 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:03.943250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818050-h78xl" event={"ID":"a643cba3-f7d7-4070-a8d9-df6cf2f54445","Type":"ContainerDied","Data":"4ac4d79b5cc12362da1a2a8d200e0c3b41c637375e5b5b9910bf9ab6e32f9319"}
Sep 10 23:30:03.943407 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:03.943296    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ac4d79b5cc12362da1a2a8d200e0c3b41c637375e5b5b9910bf9ab6e32f9319"
Sep 10 23:30:07.854845 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:07.854805    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7846-5f3927"]
Sep 10 23:30:07.855315 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:07.855173    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a643cba3-f7d7-4070-a8d9-df6cf2f54445" containerName="cleanup"
Sep 10 23:30:07.855315 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:07.855184    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a643cba3-f7d7-4070-a8d9-df6cf2f54445" containerName="cleanup"
Sep 10 23:30:07.855315 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:07.855207    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8819aba5-4867-4d59-bad2-b74f8c9c2ed2" containerName="cleanup"
Sep 10 23:30:07.855315 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:07.855212    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8819aba5-4867-4d59-bad2-b74f8c9c2ed2" containerName="cleanup"
Sep 10 23:30:07.855315 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:07.855275    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a643cba3-f7d7-4070-a8d9-df6cf2f54445" containerName="cleanup"
Sep 10 23:30:07.855315 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:07.855286    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8819aba5-4867-4d59-bad2-b74f8c9c2ed2" containerName="cleanup"
Sep 10 23:30:07.858471 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:07.858455    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7846-5f3927"
Sep 10 23:30:07.860673 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:07.860648    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f4p5h\""
Sep 10 23:30:07.862623 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:07.862598    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7846-5f3927"]
Sep 10 23:30:07.955202 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:07.955161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdddb\" (UniqueName: \"kubernetes.io/projected/7e95c657-ab92-4377-a81b-50ab7b85c265-kube-api-access-cdddb\") pod \"test-curl-7846-5f3927\" (UID: \"7e95c657-ab92-4377-a81b-50ab7b85c265\") " pod="opendatahub/test-curl-7846-5f3927"
Sep 10 23:30:08.055925 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:08.055893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdddb\" (UniqueName: \"kubernetes.io/projected/7e95c657-ab92-4377-a81b-50ab7b85c265-kube-api-access-cdddb\") pod \"test-curl-7846-5f3927\" (UID: \"7e95c657-ab92-4377-a81b-50ab7b85c265\") " pod="opendatahub/test-curl-7846-5f3927"
Sep 10 23:30:08.062723 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:08.062702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdddb\" (UniqueName: \"kubernetes.io/projected/7e95c657-ab92-4377-a81b-50ab7b85c265-kube-api-access-cdddb\") pod \"test-curl-7846-5f3927\" (UID: \"7e95c657-ab92-4377-a81b-50ab7b85c265\") " pod="opendatahub/test-curl-7846-5f3927"
Sep 10 23:30:08.169564 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:08.169477    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7846-5f3927"
Sep 10 23:30:08.436785 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:08.436706    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7846-5f3927"]
Sep 10 23:30:08.440009 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:30:08.439971    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e95c657_ab92_4377_a81b_50ab7b85c265.slice/crio-feffc0e75928be485ac1f1300d2dc151870d2dbe31d8ad8301cf9adc87e83ad8 WatchSource:0}: Error finding container feffc0e75928be485ac1f1300d2dc151870d2dbe31d8ad8301cf9adc87e83ad8: Status 404 returned error can't find the container with id feffc0e75928be485ac1f1300d2dc151870d2dbe31d8ad8301cf9adc87e83ad8
Sep 10 23:30:08.969789 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:08.969755    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7846-5f3927" event={"ID":"7e95c657-ab92-4377-a81b-50ab7b85c265","Type":"ContainerStarted","Data":"0f4a5ef3dea501fe96052edf933b15da9407adcbf48fbe6d76d0ae4d7c0c7d77"}
Sep 10 23:30:08.969789 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:08.969798    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7846-5f3927" event={"ID":"7e95c657-ab92-4377-a81b-50ab7b85c265","Type":"ContainerStarted","Data":"feffc0e75928be485ac1f1300d2dc151870d2dbe31d8ad8301cf9adc87e83ad8"}
Sep 10 23:30:08.984377 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:08.984321    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7846-5f3927" podStartSLOduration=1.984303539 podStartE2EDuration="1.984303539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:30:08.982346828 +0000 UTC m=+1729.673924013" watchObservedRunningTime="2026-09-10 23:30:08.984303539 +0000 UTC m=+1729.675880744"
Sep 10 23:30:09.558242 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:09.558185    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7846-5f3927"]
Sep 10 23:30:09.560510 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:09.560473    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7846-5f3927"]
Sep 10 23:30:09.974579 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:30:09.974547    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7846-5f3927" container="test-curl-7846-5f3927"
Sep 10 23:30:10.980059 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:10.980017    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7846-5f3927" podUID="7e95c657-ab92-4377-a81b-50ab7b85c265" containerName="test-curl-7846-5f3927" containerID="cri-o://0f4a5ef3dea501fe96052edf933b15da9407adcbf48fbe6d76d0ae4d7c0c7d77" gracePeriod=2
Sep 10 23:30:13.165055 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:13.165029    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7846-5f3927"
Sep 10 23:30:13.307262 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:13.307157    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdddb\" (UniqueName: \"kubernetes.io/projected/7e95c657-ab92-4377-a81b-50ab7b85c265-kube-api-access-cdddb\") pod \"7e95c657-ab92-4377-a81b-50ab7b85c265\" (UID: \"7e95c657-ab92-4377-a81b-50ab7b85c265\") "
Sep 10 23:30:13.309325 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:13.309287    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e95c657-ab92-4377-a81b-50ab7b85c265-kube-api-access-cdddb" (OuterVolumeSpecName: "kube-api-access-cdddb") pod "7e95c657-ab92-4377-a81b-50ab7b85c265" (UID: "7e95c657-ab92-4377-a81b-50ab7b85c265"). InnerVolumeSpecName "kube-api-access-cdddb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:30:13.408377 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:13.408343    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdddb\" (UniqueName: \"kubernetes.io/projected/7e95c657-ab92-4377-a81b-50ab7b85c265-kube-api-access-cdddb\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:30:13.822037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:13.821998    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e95c657-ab92-4377-a81b-50ab7b85c265" path="/var/lib/kubelet/pods/7e95c657-ab92-4377-a81b-50ab7b85c265/volumes"
Sep 10 23:30:13.961818 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:13.961785    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"]
Sep 10 23:30:13.997403 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:13.997368    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e95c657-ab92-4377-a81b-50ab7b85c265" containerID="0f4a5ef3dea501fe96052edf933b15da9407adcbf48fbe6d76d0ae4d7c0c7d77" exitCode=137
Sep 10 23:30:13.997403 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:13.997405    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7846-5f3927"
Sep 10 23:30:13.997625 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:13.997443    2563 scope.go:117] "RemoveContainer" containerID="0f4a5ef3dea501fe96052edf933b15da9407adcbf48fbe6d76d0ae4d7c0c7d77"
Sep 10 23:30:14.006267 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:14.006248    2563 scope.go:117] "RemoveContainer" containerID="0f4a5ef3dea501fe96052edf933b15da9407adcbf48fbe6d76d0ae4d7c0c7d77"
Sep 10 23:30:14.006563 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:30:14.006543    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4a5ef3dea501fe96052edf933b15da9407adcbf48fbe6d76d0ae4d7c0c7d77\": container with ID starting with 0f4a5ef3dea501fe96052edf933b15da9407adcbf48fbe6d76d0ae4d7c0c7d77 not found: ID does not exist" containerID="0f4a5ef3dea501fe96052edf933b15da9407adcbf48fbe6d76d0ae4d7c0c7d77"
Sep 10 23:30:14.006617 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:14.006571    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4a5ef3dea501fe96052edf933b15da9407adcbf48fbe6d76d0ae4d7c0c7d77"} err="failed to get container status \"0f4a5ef3dea501fe96052edf933b15da9407adcbf48fbe6d76d0ae4d7c0c7d77\": rpc error: code = NotFound desc = could not find container \"0f4a5ef3dea501fe96052edf933b15da9407adcbf48fbe6d76d0ae4d7c0c7d77\": container with ID starting with 0f4a5ef3dea501fe96052edf933b15da9407adcbf48fbe6d76d0ae4d7c0c7d77 not found: ID does not exist"
Sep 10 23:30:20.256380 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:20.256346    2563 scope.go:117] "RemoveContainer" containerID="972b2c66c6945d2b705f054ed4185402432da9a1be155a5f09255a39a2bf9ba7"
Sep 10 23:30:28.981801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:28.981752    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs" podUID="6de5e873-9b5a-4184-aa35-f250ca59369b" containerName="main" containerID="cri-o://5229ba5fba95e3e21cb3e278cf9f4d67554300bc08fa9ed8adc095b0caf17574" gracePeriod=45
Sep 10 23:30:29.232143 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.232090    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:30:29.251072 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.250903    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-home\") pod \"6de5e873-9b5a-4184-aa35-f250ca59369b\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") "
Sep 10 23:30:29.251072 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.250959    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-dshm\") pod \"6de5e873-9b5a-4184-aa35-f250ca59369b\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") "
Sep 10 23:30:29.251072 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.251024    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de5e873-9b5a-4184-aa35-f250ca59369b-tls-certs\") pod \"6de5e873-9b5a-4184-aa35-f250ca59369b\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") "
Sep 10 23:30:29.251072 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.251058    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6p6t\" (UniqueName: \"kubernetes.io/projected/6de5e873-9b5a-4184-aa35-f250ca59369b-kube-api-access-s6p6t\") pod \"6de5e873-9b5a-4184-aa35-f250ca59369b\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") "
Sep 10 23:30:29.251372 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.251117    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-model-cache\") pod \"6de5e873-9b5a-4184-aa35-f250ca59369b\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") "
Sep 10 23:30:29.251372 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.251142    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-home" (OuterVolumeSpecName: "home") pod "6de5e873-9b5a-4184-aa35-f250ca59369b" (UID: "6de5e873-9b5a-4184-aa35-f250ca59369b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:30:29.251372 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.251161    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-tmp-dir\") pod \"6de5e873-9b5a-4184-aa35-f250ca59369b\" (UID: \"6de5e873-9b5a-4184-aa35-f250ca59369b\") "
Sep 10 23:30:29.251544 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.251512    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:30:29.251883 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.251710    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6de5e873-9b5a-4184-aa35-f250ca59369b" (UID: "6de5e873-9b5a-4184-aa35-f250ca59369b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:30:29.252075 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.252044    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-model-cache" (OuterVolumeSpecName: "model-cache") pod "6de5e873-9b5a-4184-aa35-f250ca59369b" (UID: "6de5e873-9b5a-4184-aa35-f250ca59369b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:30:29.253801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.253775    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de5e873-9b5a-4184-aa35-f250ca59369b-kube-api-access-s6p6t" (OuterVolumeSpecName: "kube-api-access-s6p6t") pod "6de5e873-9b5a-4184-aa35-f250ca59369b" (UID: "6de5e873-9b5a-4184-aa35-f250ca59369b"). InnerVolumeSpecName "kube-api-access-s6p6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:30:29.254477 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.254453    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-dshm" (OuterVolumeSpecName: "dshm") pod "6de5e873-9b5a-4184-aa35-f250ca59369b" (UID: "6de5e873-9b5a-4184-aa35-f250ca59369b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:30:29.254611 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.254554    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de5e873-9b5a-4184-aa35-f250ca59369b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6de5e873-9b5a-4184-aa35-f250ca59369b" (UID: "6de5e873-9b5a-4184-aa35-f250ca59369b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:30:29.352438 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.352389    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:30:29.352438 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.352433    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:30:29.352438 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.352442    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6de5e873-9b5a-4184-aa35-f250ca59369b-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:30:29.352438 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.352449    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6de5e873-9b5a-4184-aa35-f250ca59369b-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:30:29.352438 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:29.352458    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6p6t\" (UniqueName: \"kubernetes.io/projected/6de5e873-9b5a-4184-aa35-f250ca59369b-kube-api-access-s6p6t\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:30:30.083913 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:30.083826    2563 generic.go:356] "Generic (PLEG): container finished" podID="6de5e873-9b5a-4184-aa35-f250ca59369b" containerID="5229ba5fba95e3e21cb3e278cf9f4d67554300bc08fa9ed8adc095b0caf17574" exitCode=0
Sep 10 23:30:30.083913 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:30.083869    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"
Sep 10 23:30:30.084359 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:30.083916    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs" event={"ID":"6de5e873-9b5a-4184-aa35-f250ca59369b","Type":"ContainerDied","Data":"5229ba5fba95e3e21cb3e278cf9f4d67554300bc08fa9ed8adc095b0caf17574"}
Sep 10 23:30:30.084359 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:30.083958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs" event={"ID":"6de5e873-9b5a-4184-aa35-f250ca59369b","Type":"ContainerDied","Data":"82e68691345a2f2488e528d6773048ef1b574204adf26020ebfe103091e94cf9"}
Sep 10 23:30:30.084359 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:30.083979    2563 scope.go:117] "RemoveContainer" containerID="5229ba5fba95e3e21cb3e278cf9f4d67554300bc08fa9ed8adc095b0caf17574"
Sep 10 23:30:30.092745 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:30.092724    2563 scope.go:117] "RemoveContainer" containerID="5229ba5fba95e3e21cb3e278cf9f4d67554300bc08fa9ed8adc095b0caf17574"
Sep 10 23:30:30.093027 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:30:30.093008    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5229ba5fba95e3e21cb3e278cf9f4d67554300bc08fa9ed8adc095b0caf17574\": container with ID starting with 5229ba5fba95e3e21cb3e278cf9f4d67554300bc08fa9ed8adc095b0caf17574 not found: ID does not exist" containerID="5229ba5fba95e3e21cb3e278cf9f4d67554300bc08fa9ed8adc095b0caf17574"
Sep 10 23:30:30.093103 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:30.093038    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5229ba5fba95e3e21cb3e278cf9f4d67554300bc08fa9ed8adc095b0caf17574"} err="failed to get container status \"5229ba5fba95e3e21cb3e278cf9f4d67554300bc08fa9ed8adc095b0caf17574\": rpc error: code = NotFound desc = could not find container \"5229ba5fba95e3e21cb3e278cf9f4d67554300bc08fa9ed8adc095b0caf17574\": container with ID starting with 5229ba5fba95e3e21cb3e278cf9f4d67554300bc08fa9ed8adc095b0caf17574 not found: ID does not exist"
Sep 10 23:30:30.099409 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:30.099383    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"]
Sep 10 23:30:30.102912 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:30.102886    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/sub-test-model-83be66-kserve-d8fdb4d9f-875fs"]
Sep 10 23:30:31.814375 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:31.814344    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6de5e873-9b5a-4184-aa35-f250ca59369b" path="/var/lib/kubelet/pods/6de5e873-9b5a-4184-aa35-f250ca59369b/volumes"
Sep 10 23:30:32.354844 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.354812    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"]
Sep 10 23:30:32.355299 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.355265    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e95c657-ab92-4377-a81b-50ab7b85c265" containerName="test-curl-7846-5f3927"
Sep 10 23:30:32.355299 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.355288    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e95c657-ab92-4377-a81b-50ab7b85c265" containerName="test-curl-7846-5f3927"
Sep 10 23:30:32.355515 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.355337    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de5e873-9b5a-4184-aa35-f250ca59369b" containerName="main"
Sep 10 23:30:32.355515 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.355346    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de5e873-9b5a-4184-aa35-f250ca59369b" containerName="main"
Sep 10 23:30:32.355515 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.355416    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e95c657-ab92-4377-a81b-50ab7b85c265" containerName="test-curl-7846-5f3927"
Sep 10 23:30:32.355515 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.355432    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6de5e873-9b5a-4184-aa35-f250ca59369b" containerName="main"
Sep 10 23:30:32.360201 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.360182    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.362701 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.362671    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6c97f2\"/\"kube-root-ca.crt\""
Sep 10 23:30:32.363398 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.363368    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6c97f2\"/\"default-dockercfg-vxzdz\""
Sep 10 23:30:32.363535 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.363405    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6c97f2\"/\"openshift-service-ca.crt\""
Sep 10 23:30:32.363535 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.363420    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6c97f2\"/\"rate-test-model-6c97f2-kserve-self-signed-certs\""
Sep 10 23:30:32.367854 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.367825    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"]
Sep 10 23:30:32.483009 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.482973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-home\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.483169 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.483023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-model-cache\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.483169 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.483046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvzwg\" (UniqueName: \"kubernetes.io/projected/80f6a6af-d163-42a7-9dfe-72a72de34aa6-kube-api-access-fvzwg\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.483169 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.483108    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-dshm\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.483169 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.483159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f6a6af-d163-42a7-9dfe-72a72de34aa6-tls-certs\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.483333 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.483182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-tmp-dir\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.584460 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.584412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-home\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.584460 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.584468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-model-cache\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.584708 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.584523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvzwg\" (UniqueName: \"kubernetes.io/projected/80f6a6af-d163-42a7-9dfe-72a72de34aa6-kube-api-access-fvzwg\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.584708 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.584549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-dshm\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.584708 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.584589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f6a6af-d163-42a7-9dfe-72a72de34aa6-tls-certs\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.584708 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.584619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-tmp-dir\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.585024 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.584986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-home\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.585024 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.585000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-model-cache\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.585190 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.585027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-tmp-dir\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.586818 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.586799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-dshm\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.587118 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.587103    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f6a6af-d163-42a7-9dfe-72a72de34aa6-tls-certs\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.591920 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.591898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvzwg\" (UniqueName: \"kubernetes.io/projected/80f6a6af-d163-42a7-9dfe-72a72de34aa6-kube-api-access-fvzwg\") pod \"rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") " pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.671571 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.671439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:32.946004 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:32.945928    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"]
Sep 10 23:30:32.949345 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:30:32.949320    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80f6a6af_d163_42a7_9dfe_72a72de34aa6.slice/crio-3e13162211035f7c77e3e1de6fbca06c401b701645cc4ed33591543869d2255e WatchSource:0}: Error finding container 3e13162211035f7c77e3e1de6fbca06c401b701645cc4ed33591543869d2255e: Status 404 returned error can't find the container with id 3e13162211035f7c77e3e1de6fbca06c401b701645cc4ed33591543869d2255e
Sep 10 23:30:33.101670 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:33.101629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9" event={"ID":"80f6a6af-d163-42a7-9dfe-72a72de34aa6","Type":"ContainerStarted","Data":"ab9ce0498f2d05cde2b2b78504072569e4408e0bd28cc02e5a7f43c1098bf858"}
Sep 10 23:30:33.101844 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:33.101685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9" event={"ID":"80f6a6af-d163-42a7-9dfe-72a72de34aa6","Type":"ContainerStarted","Data":"3e13162211035f7c77e3e1de6fbca06c401b701645cc4ed33591543869d2255e"}
Sep 10 23:30:33.118206 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:33.118099    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9" podStartSLOduration=1.118079392 podStartE2EDuration="1.118079392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:30:33.117449177 +0000 UTC m=+1753.809026382" watchObservedRunningTime="2026-09-10 23:30:33.118079392 +0000 UTC m=+1753.809656579"
Sep 10 23:30:42.671914 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:42.671875    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:42.672415 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:42.671931    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:42.684305 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:42.684278    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:30:43.165423 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:30:43.165395    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:31:10.256570 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.256534    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"]
Sep 10 23:31:10.260320 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.260296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.263598 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.263574    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-83be66\"/\"default-dockercfg-dq6xc\""
Sep 10 23:31:10.263737 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.263606    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-83be66\"/\"openshift-service-ca.crt\""
Sep 10 23:31:10.263737 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.263592    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-83be66\"/\"rate-test-model-83be66-kserve-self-signed-certs\""
Sep 10 23:31:10.263737 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.263599    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-83be66\"/\"kube-root-ca.crt\""
Sep 10 23:31:10.274295 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.271771    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"]
Sep 10 23:31:10.336224 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.336193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-model-cache\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.336388 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.336234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee7b87e-a88b-4c28-811d-46eb2217fa37-tls-certs\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.336388 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.336287    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tcf9\" (UniqueName: \"kubernetes.io/projected/cee7b87e-a88b-4c28-811d-46eb2217fa37-kube-api-access-4tcf9\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.336388 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.336310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-tmp-dir\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.336388 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.336330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-home\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.336576 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.336404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-dshm\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.437576 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.437531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-model-cache\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.437756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.437592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee7b87e-a88b-4c28-811d-46eb2217fa37-tls-certs\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.437756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.437653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tcf9\" (UniqueName: \"kubernetes.io/projected/cee7b87e-a88b-4c28-811d-46eb2217fa37-kube-api-access-4tcf9\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.437756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.437687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-tmp-dir\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.437756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.437713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-home\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.437756 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.437737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-dshm\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.438001 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.437934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-model-cache\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.438056 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.438012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-tmp-dir\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.438056 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.438041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-home\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.439992 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.439972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-dshm\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.440328 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.440305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee7b87e-a88b-4c28-811d-46eb2217fa37-tls-certs\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.444721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.444703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tcf9\" (UniqueName: \"kubernetes.io/projected/cee7b87e-a88b-4c28-811d-46eb2217fa37-kube-api-access-4tcf9\") pod \"rate-test-model-83be66-kserve-7cc875fb87-wks7x\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") " pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.576933 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.576897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:10.845927 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:10.845899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"]
Sep 10 23:31:10.848557 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:31:10.848527    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcee7b87e_a88b_4c28_811d_46eb2217fa37.slice/crio-535841cd0f4f36b53397e3fa3ac4f9ec226cd0f0b0405badc79939278c0a5102 WatchSource:0}: Error finding container 535841cd0f4f36b53397e3fa3ac4f9ec226cd0f0b0405badc79939278c0a5102: Status 404 returned error can't find the container with id 535841cd0f4f36b53397e3fa3ac4f9ec226cd0f0b0405badc79939278c0a5102
Sep 10 23:31:11.298960 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:11.298853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x" event={"ID":"cee7b87e-a88b-4c28-811d-46eb2217fa37","Type":"ContainerStarted","Data":"caae5351cd22ca1d8ebf3233cb134643152635871a6f53fbc66be3e3061e8878"}
Sep 10 23:31:11.298960 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:11.298899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x" event={"ID":"cee7b87e-a88b-4c28-811d-46eb2217fa37","Type":"ContainerStarted","Data":"535841cd0f4f36b53397e3fa3ac4f9ec226cd0f0b0405badc79939278c0a5102"}
Sep 10 23:31:11.316575 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:11.316529    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x" podStartSLOduration=1.316514363 podStartE2EDuration="1.316514363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:31:11.314081547 +0000 UTC m=+1792.005658751" watchObservedRunningTime="2026-09-10 23:31:11.316514363 +0000 UTC m=+1792.008091540"
Sep 10 23:31:19.926369 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:19.926344    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:31:19.929239 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:19.929218    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:31:20.577309 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:20.577281    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:20.577531 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:20.577350    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:20.589607 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:20.589585    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:21.367856 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:21.367826    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:42.830171 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:42.830132    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"]
Sep 10 23:31:43.767225 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:43.767192    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"]
Sep 10 23:31:57.849309 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:57.849270    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9" podUID="80f6a6af-d163-42a7-9dfe-72a72de34aa6" containerName="main" containerID="cri-o://ab9ce0498f2d05cde2b2b78504072569e4408e0bd28cc02e5a7f43c1098bf858" gracePeriod=45
Sep 10 23:31:58.106877 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.106854    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:31:58.178721 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.178688    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvzwg\" (UniqueName: \"kubernetes.io/projected/80f6a6af-d163-42a7-9dfe-72a72de34aa6-kube-api-access-fvzwg\") pod \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") "
Sep 10 23:31:58.178900 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.178741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-model-cache\") pod \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") "
Sep 10 23:31:58.178900 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.178774    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-home\") pod \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") "
Sep 10 23:31:58.178900 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.178809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f6a6af-d163-42a7-9dfe-72a72de34aa6-tls-certs\") pod \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") "
Sep 10 23:31:58.179061 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.178907    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-dshm\") pod \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") "
Sep 10 23:31:58.179061 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.178932    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-tmp-dir\") pod \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\" (UID: \"80f6a6af-d163-42a7-9dfe-72a72de34aa6\") "
Sep 10 23:31:58.179061 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.179040    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-home" (OuterVolumeSpecName: "home") pod "80f6a6af-d163-42a7-9dfe-72a72de34aa6" (UID: "80f6a6af-d163-42a7-9dfe-72a72de34aa6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:31:58.179224 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.179047    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-model-cache" (OuterVolumeSpecName: "model-cache") pod "80f6a6af-d163-42a7-9dfe-72a72de34aa6" (UID: "80f6a6af-d163-42a7-9dfe-72a72de34aa6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:31:58.179301 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.179278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80f6a6af-d163-42a7-9dfe-72a72de34aa6" (UID: "80f6a6af-d163-42a7-9dfe-72a72de34aa6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:31:58.179357 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.179301    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:31:58.179357 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.179320    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:31:58.180891 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.180860    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-dshm" (OuterVolumeSpecName: "dshm") pod "80f6a6af-d163-42a7-9dfe-72a72de34aa6" (UID: "80f6a6af-d163-42a7-9dfe-72a72de34aa6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:31:58.180891 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.180880    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f6a6af-d163-42a7-9dfe-72a72de34aa6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80f6a6af-d163-42a7-9dfe-72a72de34aa6" (UID: "80f6a6af-d163-42a7-9dfe-72a72de34aa6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:31:58.181039 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.180963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f6a6af-d163-42a7-9dfe-72a72de34aa6-kube-api-access-fvzwg" (OuterVolumeSpecName: "kube-api-access-fvzwg") pod "80f6a6af-d163-42a7-9dfe-72a72de34aa6" (UID: "80f6a6af-d163-42a7-9dfe-72a72de34aa6"). InnerVolumeSpecName "kube-api-access-fvzwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:31:58.280705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.280662    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f6a6af-d163-42a7-9dfe-72a72de34aa6-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:31:58.280705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.280698    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:31:58.280705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.280708    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f6a6af-d163-42a7-9dfe-72a72de34aa6-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:31:58.280705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.280716    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvzwg\" (UniqueName: \"kubernetes.io/projected/80f6a6af-d163-42a7-9dfe-72a72de34aa6-kube-api-access-fvzwg\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:31:58.554321 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.554231    2563 generic.go:356] "Generic (PLEG): container finished" podID="80f6a6af-d163-42a7-9dfe-72a72de34aa6" containerID="ab9ce0498f2d05cde2b2b78504072569e4408e0bd28cc02e5a7f43c1098bf858" exitCode=0
Sep 10 23:31:58.554321 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.554274    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"
Sep 10 23:31:58.554588 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.554318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9" event={"ID":"80f6a6af-d163-42a7-9dfe-72a72de34aa6","Type":"ContainerDied","Data":"ab9ce0498f2d05cde2b2b78504072569e4408e0bd28cc02e5a7f43c1098bf858"}
Sep 10 23:31:58.554588 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.554366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9" event={"ID":"80f6a6af-d163-42a7-9dfe-72a72de34aa6","Type":"ContainerDied","Data":"3e13162211035f7c77e3e1de6fbca06c401b701645cc4ed33591543869d2255e"}
Sep 10 23:31:58.554588 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.554389    2563 scope.go:117] "RemoveContainer" containerID="ab9ce0498f2d05cde2b2b78504072569e4408e0bd28cc02e5a7f43c1098bf858"
Sep 10 23:31:58.564418 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.564401    2563 scope.go:117] "RemoveContainer" containerID="ab9ce0498f2d05cde2b2b78504072569e4408e0bd28cc02e5a7f43c1098bf858"
Sep 10 23:31:58.564686 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:31:58.564665    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9ce0498f2d05cde2b2b78504072569e4408e0bd28cc02e5a7f43c1098bf858\": container with ID starting with ab9ce0498f2d05cde2b2b78504072569e4408e0bd28cc02e5a7f43c1098bf858 not found: ID does not exist" containerID="ab9ce0498f2d05cde2b2b78504072569e4408e0bd28cc02e5a7f43c1098bf858"
Sep 10 23:31:58.564755 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.564697    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9ce0498f2d05cde2b2b78504072569e4408e0bd28cc02e5a7f43c1098bf858"} err="failed to get container status \"ab9ce0498f2d05cde2b2b78504072569e4408e0bd28cc02e5a7f43c1098bf858\": rpc error: code = NotFound desc = could not find container \"ab9ce0498f2d05cde2b2b78504072569e4408e0bd28cc02e5a7f43c1098bf858\": container with ID starting with ab9ce0498f2d05cde2b2b78504072569e4408e0bd28cc02e5a7f43c1098bf858 not found: ID does not exist"
Sep 10 23:31:58.577044 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.577022    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"]
Sep 10 23:31:58.585646 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.585621    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6c97f2/rate-test-model-6c97f2-kserve-dfdd546c9-v2wm9"]
Sep 10 23:31:58.787935 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:58.787905    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x" podUID="cee7b87e-a88b-4c28-811d-46eb2217fa37" containerName="main" containerID="cri-o://caae5351cd22ca1d8ebf3233cb134643152635871a6f53fbc66be3e3061e8878" gracePeriod=45
Sep 10 23:31:59.038168 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.038145    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:59.086781 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.086754    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tcf9\" (UniqueName: \"kubernetes.io/projected/cee7b87e-a88b-4c28-811d-46eb2217fa37-kube-api-access-4tcf9\") pod \"cee7b87e-a88b-4c28-811d-46eb2217fa37\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") "
Sep 10 23:31:59.086943 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.086793    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee7b87e-a88b-4c28-811d-46eb2217fa37-tls-certs\") pod \"cee7b87e-a88b-4c28-811d-46eb2217fa37\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") "
Sep 10 23:31:59.086943 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.086826    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-tmp-dir\") pod \"cee7b87e-a88b-4c28-811d-46eb2217fa37\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") "
Sep 10 23:31:59.086943 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.086903    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-dshm\") pod \"cee7b87e-a88b-4c28-811d-46eb2217fa37\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") "
Sep 10 23:31:59.086943 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.086934    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-model-cache\") pod \"cee7b87e-a88b-4c28-811d-46eb2217fa37\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") "
Sep 10 23:31:59.087169 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.086955    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-home\") pod \"cee7b87e-a88b-4c28-811d-46eb2217fa37\" (UID: \"cee7b87e-a88b-4c28-811d-46eb2217fa37\") "
Sep 10 23:31:59.087169 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.087128    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cee7b87e-a88b-4c28-811d-46eb2217fa37" (UID: "cee7b87e-a88b-4c28-811d-46eb2217fa37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:31:59.087282 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.087234    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-model-cache" (OuterVolumeSpecName: "model-cache") pod "cee7b87e-a88b-4c28-811d-46eb2217fa37" (UID: "cee7b87e-a88b-4c28-811d-46eb2217fa37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:31:59.087337 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.087321    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-home" (OuterVolumeSpecName: "home") pod "cee7b87e-a88b-4c28-811d-46eb2217fa37" (UID: "cee7b87e-a88b-4c28-811d-46eb2217fa37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:31:59.087406 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.087387    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:31:59.087448 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.087412    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:31:59.088939 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.088917    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee7b87e-a88b-4c28-811d-46eb2217fa37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cee7b87e-a88b-4c28-811d-46eb2217fa37" (UID: "cee7b87e-a88b-4c28-811d-46eb2217fa37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:31:59.089048 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.089025    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-dshm" (OuterVolumeSpecName: "dshm") pod "cee7b87e-a88b-4c28-811d-46eb2217fa37" (UID: "cee7b87e-a88b-4c28-811d-46eb2217fa37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:31:59.089112 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.089063    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee7b87e-a88b-4c28-811d-46eb2217fa37-kube-api-access-4tcf9" (OuterVolumeSpecName: "kube-api-access-4tcf9") pod "cee7b87e-a88b-4c28-811d-46eb2217fa37" (UID: "cee7b87e-a88b-4c28-811d-46eb2217fa37"). InnerVolumeSpecName "kube-api-access-4tcf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:31:59.188513 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.188439    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:31:59.188513 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.188463    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cee7b87e-a88b-4c28-811d-46eb2217fa37-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:31:59.188513 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.188472    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tcf9\" (UniqueName: \"kubernetes.io/projected/cee7b87e-a88b-4c28-811d-46eb2217fa37-kube-api-access-4tcf9\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:31:59.188513 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.188482    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cee7b87e-a88b-4c28-811d-46eb2217fa37-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:31:59.561876 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.561844    2563 generic.go:356] "Generic (PLEG): container finished" podID="cee7b87e-a88b-4c28-811d-46eb2217fa37" containerID="caae5351cd22ca1d8ebf3233cb134643152635871a6f53fbc66be3e3061e8878" exitCode=0
Sep 10 23:31:59.562037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.561885    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"
Sep 10 23:31:59.562037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.561911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x" event={"ID":"cee7b87e-a88b-4c28-811d-46eb2217fa37","Type":"ContainerDied","Data":"caae5351cd22ca1d8ebf3233cb134643152635871a6f53fbc66be3e3061e8878"}
Sep 10 23:31:59.562037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.561941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x" event={"ID":"cee7b87e-a88b-4c28-811d-46eb2217fa37","Type":"ContainerDied","Data":"535841cd0f4f36b53397e3fa3ac4f9ec226cd0f0b0405badc79939278c0a5102"}
Sep 10 23:31:59.562037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.561957    2563 scope.go:117] "RemoveContainer" containerID="caae5351cd22ca1d8ebf3233cb134643152635871a6f53fbc66be3e3061e8878"
Sep 10 23:31:59.571348 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.571330    2563 scope.go:117] "RemoveContainer" containerID="caae5351cd22ca1d8ebf3233cb134643152635871a6f53fbc66be3e3061e8878"
Sep 10 23:31:59.571611 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:31:59.571593    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caae5351cd22ca1d8ebf3233cb134643152635871a6f53fbc66be3e3061e8878\": container with ID starting with caae5351cd22ca1d8ebf3233cb134643152635871a6f53fbc66be3e3061e8878 not found: ID does not exist" containerID="caae5351cd22ca1d8ebf3233cb134643152635871a6f53fbc66be3e3061e8878"
Sep 10 23:31:59.571695 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.571622    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caae5351cd22ca1d8ebf3233cb134643152635871a6f53fbc66be3e3061e8878"} err="failed to get container status \"caae5351cd22ca1d8ebf3233cb134643152635871a6f53fbc66be3e3061e8878\": rpc error: code = NotFound desc = could not find container \"caae5351cd22ca1d8ebf3233cb134643152635871a6f53fbc66be3e3061e8878\": container with ID starting with caae5351cd22ca1d8ebf3233cb134643152635871a6f53fbc66be3e3061e8878 not found: ID does not exist"
Sep 10 23:31:59.581979 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.581959    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"]
Sep 10 23:31:59.584869 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.584847    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-83be66/rate-test-model-83be66-kserve-7cc875fb87-wks7x"]
Sep 10 23:31:59.813830 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.813751    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f6a6af-d163-42a7-9dfe-72a72de34aa6" path="/var/lib/kubelet/pods/80f6a6af-d163-42a7-9dfe-72a72de34aa6/volumes"
Sep 10 23:31:59.814220 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:31:59.814200    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cee7b87e-a88b-4c28-811d-46eb2217fa37" path="/var/lib/kubelet/pods/cee7b87e-a88b-4c28-811d-46eb2217fa37/volumes"
Sep 10 23:32:18.897400 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:18.897369    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"]
Sep 10 23:32:18.897885 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:18.897760    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cee7b87e-a88b-4c28-811d-46eb2217fa37" containerName="main"
Sep 10 23:32:18.897885 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:18.897772    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cee7b87e-a88b-4c28-811d-46eb2217fa37" containerName="main"
Sep 10 23:32:18.897885 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:18.897792    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f6a6af-d163-42a7-9dfe-72a72de34aa6" containerName="main"
Sep 10 23:32:18.897885 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:18.897798    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f6a6af-d163-42a7-9dfe-72a72de34aa6" containerName="main"
Sep 10 23:32:18.897885 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:18.897854    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cee7b87e-a88b-4c28-811d-46eb2217fa37" containerName="main"
Sep 10 23:32:18.897885 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:18.897866    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f6a6af-d163-42a7-9dfe-72a72de34aa6" containerName="main"
Sep 10 23:32:18.902145 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:18.902129    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"
Sep 10 23:32:18.904552 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:18.904529    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-706b9d-dockercfg-p5jlk\""
Sep 10 23:32:18.904678 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:18.904643    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-706b9d\""
Sep 10 23:32:18.912159 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:18.912130    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"]
Sep 10 23:32:18.913861 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:18.913837    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"]
Sep 10 23:32:18.917312 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:18.917294    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"
Sep 10 23:32:18.927991 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:18.927957    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"]
Sep 10 23:32:19.074544 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.074484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74abccaf-3996-427d-9e3d-04d0c9480645-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c\" (UID: \"74abccaf-3996-427d-9e3d-04d0c9480645\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"
Sep 10 23:32:19.074705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.074615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711d62f2-2ef7-4137-a1ea-a9234bca6b02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2\" (UID: \"711d62f2-2ef7-4137-a1ea-a9234bca6b02\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"
Sep 10 23:32:19.074705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.074643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rcwd\" (UniqueName: \"kubernetes.io/projected/711d62f2-2ef7-4137-a1ea-a9234bca6b02-kube-api-access-5rcwd\") pod \"payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2\" (UID: \"711d62f2-2ef7-4137-a1ea-a9234bca6b02\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"
Sep 10 23:32:19.074705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.074671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92v22\" (UniqueName: \"kubernetes.io/projected/74abccaf-3996-427d-9e3d-04d0c9480645-kube-api-access-92v22\") pod \"payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c\" (UID: \"74abccaf-3996-427d-9e3d-04d0c9480645\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"
Sep 10 23:32:19.175993 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.175906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rcwd\" (UniqueName: \"kubernetes.io/projected/711d62f2-2ef7-4137-a1ea-a9234bca6b02-kube-api-access-5rcwd\") pod \"payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2\" (UID: \"711d62f2-2ef7-4137-a1ea-a9234bca6b02\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"
Sep 10 23:32:19.175993 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.175963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92v22\" (UniqueName: \"kubernetes.io/projected/74abccaf-3996-427d-9e3d-04d0c9480645-kube-api-access-92v22\") pod \"payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c\" (UID: \"74abccaf-3996-427d-9e3d-04d0c9480645\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"
Sep 10 23:32:19.175993 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.175995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74abccaf-3996-427d-9e3d-04d0c9480645-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c\" (UID: \"74abccaf-3996-427d-9e3d-04d0c9480645\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"
Sep 10 23:32:19.176293 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.176074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711d62f2-2ef7-4137-a1ea-a9234bca6b02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2\" (UID: \"711d62f2-2ef7-4137-a1ea-a9234bca6b02\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"
Sep 10 23:32:19.176837 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.176813    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711d62f2-2ef7-4137-a1ea-a9234bca6b02-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2\" (UID: \"711d62f2-2ef7-4137-a1ea-a9234bca6b02\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"
Sep 10 23:32:19.176938 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.176817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74abccaf-3996-427d-9e3d-04d0c9480645-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c\" (UID: \"74abccaf-3996-427d-9e3d-04d0c9480645\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"
Sep 10 23:32:19.183703 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.183677    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92v22\" (UniqueName: \"kubernetes.io/projected/74abccaf-3996-427d-9e3d-04d0c9480645-kube-api-access-92v22\") pod \"payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c\" (UID: \"74abccaf-3996-427d-9e3d-04d0c9480645\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"
Sep 10 23:32:19.183823 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.183684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rcwd\" (UniqueName: \"kubernetes.io/projected/711d62f2-2ef7-4137-a1ea-a9234bca6b02-kube-api-access-5rcwd\") pod \"payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2\" (UID: \"711d62f2-2ef7-4137-a1ea-a9234bca6b02\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"
Sep 10 23:32:19.213638 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.213612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"
Sep 10 23:32:19.228462 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.228442    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"
Sep 10 23:32:19.487764 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.487733    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"]
Sep 10 23:32:19.490182 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:32:19.490138    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74abccaf_3996_427d_9e3d_04d0c9480645.slice/crio-68546c35fd5f13d31cb70677e428728d8692f1723650e2bfa6de0a0a8a336bda WatchSource:0}: Error finding container 68546c35fd5f13d31cb70677e428728d8692f1723650e2bfa6de0a0a8a336bda: Status 404 returned error can't find the container with id 68546c35fd5f13d31cb70677e428728d8692f1723650e2bfa6de0a0a8a336bda
Sep 10 23:32:19.507835 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.507813    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"]
Sep 10 23:32:19.510641 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:32:19.510609    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711d62f2_2ef7_4137_a1ea_a9234bca6b02.slice/crio-10add6bb42d508fc86659393d95695dc35c04101cbeeb47770f75caea017e01e WatchSource:0}: Error finding container 10add6bb42d508fc86659393d95695dc35c04101cbeeb47770f75caea017e01e: Status 404 returned error can't find the container with id 10add6bb42d508fc86659393d95695dc35c04101cbeeb47770f75caea017e01e
Sep 10 23:32:19.666670 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.666632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2" event={"ID":"711d62f2-2ef7-4137-a1ea-a9234bca6b02","Type":"ContainerStarted","Data":"d7368e6c5851b87990ad745dcbf3a72c129cc53ccb4068a5664ac009eb3403b5"}
Sep 10 23:32:19.666839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.666690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2" event={"ID":"711d62f2-2ef7-4137-a1ea-a9234bca6b02","Type":"ContainerStarted","Data":"10add6bb42d508fc86659393d95695dc35c04101cbeeb47770f75caea017e01e"}
Sep 10 23:32:19.666839 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.666797    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"
Sep 10 23:32:19.668363 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.668339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c" event={"ID":"74abccaf-3996-427d-9e3d-04d0c9480645","Type":"ContainerStarted","Data":"e28c1e2a46c12c20f174fa8247b9be7fa95025aa7514eea585f5b95a3f013e9b"}
Sep 10 23:32:19.668513 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.668386    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"
Sep 10 23:32:19.668513 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.668405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c" event={"ID":"74abccaf-3996-427d-9e3d-04d0c9480645","Type":"ContainerStarted","Data":"68546c35fd5f13d31cb70677e428728d8692f1723650e2bfa6de0a0a8a336bda"}
Sep 10 23:32:19.684759 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.684701    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2" podStartSLOduration=1.684683245 podStartE2EDuration="1.684683245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:32:19.68197925 +0000 UTC m=+1860.373556456" watchObservedRunningTime="2026-09-10 23:32:19.684683245 +0000 UTC m=+1860.376260431"
Sep 10 23:32:19.697549 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:19.697507    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c" podStartSLOduration=1.697479393 podStartE2EDuration="1.697479393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:32:19.696056217 +0000 UTC m=+1860.387633426" watchObservedRunningTime="2026-09-10 23:32:19.697479393 +0000 UTC m=+1860.389056578"
Sep 10 23:32:30.675668 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:30.675635    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"
Sep 10 23:32:30.676156 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:30.675708    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"
Sep 10 23:32:46.952538 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:46.952441    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mcn47_a2dc4813-36ba-4324-bf13-4e83f11ded1d/payload-processing/0.log"
Sep 10 23:32:47.076275 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:47.076240    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2_711d62f2-2ef7-4137-a1ea-a9234bca6b02/payload-processing/0.log"
Sep 10 23:32:52.051325 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.051291    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"]
Sep 10 23:32:52.055218 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.055196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.058852 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.058827    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ea2b31\"/\"default-dockercfg-7ndtk\""
Sep 10 23:32:52.058991 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.058942    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ea2b31\"/\"kube-root-ca.crt\""
Sep 10 23:32:52.058991 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.058861    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ea2b31\"/\"openshift-service-ca.crt\""
Sep 10 23:32:52.059111 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.058827    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ea2b31\"/\"ipp-route-ea2b31-kserve-self-signed-certs\""
Sep 10 23:32:52.076939 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.076908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"]
Sep 10 23:32:52.179171 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.179123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41537d81-4a7a-4585-9a6c-4b9d33d4af80-tls-certs\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.179352 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.179190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-dshm\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.179352 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.179218    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-home\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.179352 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.179299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjzlr\" (UniqueName: \"kubernetes.io/projected/41537d81-4a7a-4585-9a6c-4b9d33d4af80-kube-api-access-cjzlr\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.179352 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.179333    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-model-cache\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.179520 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.179357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-tmp-dir\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.280749 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.280714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41537d81-4a7a-4585-9a6c-4b9d33d4af80-tls-certs\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.280893 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.280763    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-dshm\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.280893 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.280782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-home\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.280893 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.280830    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjzlr\" (UniqueName: \"kubernetes.io/projected/41537d81-4a7a-4585-9a6c-4b9d33d4af80-kube-api-access-cjzlr\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.280893 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.280849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-model-cache\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.280893 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.280865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-tmp-dir\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.281243 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.281220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-model-cache\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.281306 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.281248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-tmp-dir\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.281361 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.281317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-home\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.283069 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.283047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-dshm\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.283238 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.283219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41537d81-4a7a-4585-9a6c-4b9d33d4af80-tls-certs\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.288405 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.288385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjzlr\" (UniqueName: \"kubernetes.io/projected/41537d81-4a7a-4585-9a6c-4b9d33d4af80-kube-api-access-cjzlr\") pod \"ipp-route-ea2b31-kserve-7474d5b58c-wvn7b\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") " pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.374121 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.374093    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:32:52.644522 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.644477    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"]
Sep 10 23:32:52.646099 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:32:52.646072    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41537d81_4a7a_4585_9a6c_4b9d33d4af80.slice/crio-753ec7d63072dcbbbb126fe6fc629717afc2b3b92fc646af2100eda98fc6dbc5 WatchSource:0}: Error finding container 753ec7d63072dcbbbb126fe6fc629717afc2b3b92fc646af2100eda98fc6dbc5: Status 404 returned error can't find the container with id 753ec7d63072dcbbbb126fe6fc629717afc2b3b92fc646af2100eda98fc6dbc5
Sep 10 23:32:52.840117 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.840081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b" event={"ID":"41537d81-4a7a-4585-9a6c-4b9d33d4af80","Type":"ContainerStarted","Data":"eb5b6df0b8262614b70072e27619bd16832f83cdde1916cf2c264bce65393aa0"}
Sep 10 23:32:52.840117 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.840123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b" event={"ID":"41537d81-4a7a-4585-9a6c-4b9d33d4af80","Type":"ContainerStarted","Data":"753ec7d63072dcbbbb126fe6fc629717afc2b3b92fc646af2100eda98fc6dbc5"}
Sep 10 23:32:52.856885 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:32:52.856843    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b" podStartSLOduration=0.856826262 podStartE2EDuration="856.826262ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:32:52.855867917 +0000 UTC m=+1893.547445132" watchObservedRunningTime="2026-09-10 23:32:52.856826262 +0000 UTC m=+1893.548403445"
Sep 10 23:33:02.374369 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:02.374330    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:33:02.374801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:02.374384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:33:02.386805 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:02.386781    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:33:02.903532 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:02.903487    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:33:17.758229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:17.758196    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mcn47_a2dc4813-36ba-4324-bf13-4e83f11ded1d/payload-processing/0.log"
Sep 10 23:33:17.876144 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:17.876115    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2_711d62f2-2ef7-4137-a1ea-a9234bca6b02/payload-processing/0.log"
Sep 10 23:33:20.312617 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:20.312583    2563 scope.go:117] "RemoveContainer" containerID="f8123ff4f11754f1e9807669ffc56f9568d14a08bd5ae44f4c47ba671bd0a2d7"
Sep 10 23:33:20.325641 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:20.325618    2563 scope.go:117] "RemoveContainer" containerID="644c0248f7c57018d9528b17acb1a40378ad9aa8684a1d1b0f61919b5253934f"
Sep 10 23:33:23.333459 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.333420    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"]
Sep 10 23:33:23.342481 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.342457    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"
Sep 10 23:33:23.345203 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.345178    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-230464\""
Sep 10 23:33:23.345831 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.345809    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-230464-dockercfg-zhd5r\""
Sep 10 23:33:23.345965 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.345865    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"]
Sep 10 23:33:23.370330 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.370302    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"]
Sep 10 23:33:23.373695 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.373678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"
Sep 10 23:33:23.383579 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.383560    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"]
Sep 10 23:33:23.456206 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.456177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ds42\" (UniqueName: \"kubernetes.io/projected/67c2460e-404f-40fe-9805-3bafc97baf77-kube-api-access-5ds42\") pod \"payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs\" (UID: \"67c2460e-404f-40fe-9805-3bafc97baf77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"
Sep 10 23:33:23.456360 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.456220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcpfr\" (UniqueName: \"kubernetes.io/projected/9fd68b1c-8304-44fc-9110-b9bc149d17c9-kube-api-access-tcpfr\") pod \"payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm\" (UID: \"9fd68b1c-8304-44fc-9110-b9bc149d17c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"
Sep 10 23:33:23.456360 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.456242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c2460e-404f-40fe-9805-3bafc97baf77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs\" (UID: \"67c2460e-404f-40fe-9805-3bafc97baf77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"
Sep 10 23:33:23.456458 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.456377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd68b1c-8304-44fc-9110-b9bc149d17c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm\" (UID: \"9fd68b1c-8304-44fc-9110-b9bc149d17c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"
Sep 10 23:33:23.557422 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.557392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ds42\" (UniqueName: \"kubernetes.io/projected/67c2460e-404f-40fe-9805-3bafc97baf77-kube-api-access-5ds42\") pod \"payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs\" (UID: \"67c2460e-404f-40fe-9805-3bafc97baf77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"
Sep 10 23:33:23.557610 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.557436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcpfr\" (UniqueName: \"kubernetes.io/projected/9fd68b1c-8304-44fc-9110-b9bc149d17c9-kube-api-access-tcpfr\") pod \"payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm\" (UID: \"9fd68b1c-8304-44fc-9110-b9bc149d17c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"
Sep 10 23:33:23.557610 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.557461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c2460e-404f-40fe-9805-3bafc97baf77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs\" (UID: \"67c2460e-404f-40fe-9805-3bafc97baf77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"
Sep 10 23:33:23.557610 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.557530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd68b1c-8304-44fc-9110-b9bc149d17c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm\" (UID: \"9fd68b1c-8304-44fc-9110-b9bc149d17c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"
Sep 10 23:33:23.558180 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.558160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c2460e-404f-40fe-9805-3bafc97baf77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs\" (UID: \"67c2460e-404f-40fe-9805-3bafc97baf77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"
Sep 10 23:33:23.558220 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.558175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd68b1c-8304-44fc-9110-b9bc149d17c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm\" (UID: \"9fd68b1c-8304-44fc-9110-b9bc149d17c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"
Sep 10 23:33:23.565134 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.565101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ds42\" (UniqueName: \"kubernetes.io/projected/67c2460e-404f-40fe-9805-3bafc97baf77-kube-api-access-5ds42\") pod \"payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs\" (UID: \"67c2460e-404f-40fe-9805-3bafc97baf77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"
Sep 10 23:33:23.565300 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.565281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcpfr\" (UniqueName: \"kubernetes.io/projected/9fd68b1c-8304-44fc-9110-b9bc149d17c9-kube-api-access-tcpfr\") pod \"payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm\" (UID: \"9fd68b1c-8304-44fc-9110-b9bc149d17c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"
Sep 10 23:33:23.656458 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.656364    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"
Sep 10 23:33:23.686797 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.686770    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"
Sep 10 23:33:23.929689 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.929634    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"]
Sep 10 23:33:23.932100 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:33:23.932070    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c2460e_404f_40fe_9805_3bafc97baf77.slice/crio-1dcc73fd8817006daece30d42834381716f61787ca4e72df10012f3b83211ede WatchSource:0}: Error finding container 1dcc73fd8817006daece30d42834381716f61787ca4e72df10012f3b83211ede: Status 404 returned error can't find the container with id 1dcc73fd8817006daece30d42834381716f61787ca4e72df10012f3b83211ede
Sep 10 23:33:23.965182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:23.965161    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"]
Sep 10 23:33:23.967776 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:33:23.967750    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd68b1c_8304_44fc_9110_b9bc149d17c9.slice/crio-1a8ff9077990f6ad0cac2b165fa7616ec888405532f56fd78a5cf6f03a1260a1 WatchSource:0}: Error finding container 1a8ff9077990f6ad0cac2b165fa7616ec888405532f56fd78a5cf6f03a1260a1: Status 404 returned error can't find the container with id 1a8ff9077990f6ad0cac2b165fa7616ec888405532f56fd78a5cf6f03a1260a1
Sep 10 23:33:24.011517 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:24.011461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs" event={"ID":"67c2460e-404f-40fe-9805-3bafc97baf77","Type":"ContainerStarted","Data":"cfb4219ec6a7604db93ce1ad85054585b0fe709e8ef28900c573d1a601c55007"}
Sep 10 23:33:24.011638 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:24.011580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs" event={"ID":"67c2460e-404f-40fe-9805-3bafc97baf77","Type":"ContainerStarted","Data":"1dcc73fd8817006daece30d42834381716f61787ca4e72df10012f3b83211ede"}
Sep 10 23:33:24.011638 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:24.011613    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"
Sep 10 23:33:24.013010 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:24.012986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm" event={"ID":"9fd68b1c-8304-44fc-9110-b9bc149d17c9","Type":"ContainerStarted","Data":"1a8ff9077990f6ad0cac2b165fa7616ec888405532f56fd78a5cf6f03a1260a1"}
Sep 10 23:33:24.027328 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:24.027287    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs" podStartSLOduration=1.027272173 podStartE2EDuration="1.027272173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:33:24.025914912 +0000 UTC m=+1924.717492117" watchObservedRunningTime="2026-09-10 23:33:24.027272173 +0000 UTC m=+1924.718849356"
Sep 10 23:33:25.020678 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:25.020637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm" event={"ID":"9fd68b1c-8304-44fc-9110-b9bc149d17c9","Type":"ContainerStarted","Data":"dfdcdbdd2a967f9b10e57a101ca418e9c0fe110aed049a1fd79639f8a6e22d2a"}
Sep 10 23:33:25.038636 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:25.038578    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm" podStartSLOduration=2.038557886 podStartE2EDuration="2.038557886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:33:25.036733517 +0000 UTC m=+1925.728310724" watchObservedRunningTime="2026-09-10 23:33:25.038557886 +0000 UTC m=+1925.730135072"
Sep 10 23:33:26.025903 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:26.025871    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"
Sep 10 23:33:35.023396 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:35.023360    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"
Sep 10 23:33:37.033270 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:37.033240    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"
Sep 10 23:33:47.108617 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:47.108586    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk"]
Sep 10 23:33:47.112145 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:47.112125    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk"
Sep 10 23:33:47.114287 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:47.114266    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:33:47.114468 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:47.114455    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:33:47.120538 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:47.120479    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk"]
Sep 10 23:33:47.267622 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:47.267590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dca5fd-c227-41f6-aba8-48c5b9a2342e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk\" (UID: \"08dca5fd-c227-41f6-aba8-48c5b9a2342e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk"
Sep 10 23:33:47.368666 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:47.368562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dca5fd-c227-41f6-aba8-48c5b9a2342e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk\" (UID: \"08dca5fd-c227-41f6-aba8-48c5b9a2342e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk"
Sep 10 23:33:47.369232 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:47.369204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dca5fd-c227-41f6-aba8-48c5b9a2342e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk\" (UID: \"08dca5fd-c227-41f6-aba8-48c5b9a2342e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk"
Sep 10 23:33:47.425540 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:47.425484    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk"
Sep 10 23:33:47.701345 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:47.701319    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk"]
Sep 10 23:33:47.703886 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:33:47.703858    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08dca5fd_c227_41f6_aba8_48c5b9a2342e.slice/crio-82c97d356253d27b77944c260eca3a47200509c96ebac45e65313c1ec7f53b8e WatchSource:0}: Error finding container 82c97d356253d27b77944c260eca3a47200509c96ebac45e65313c1ec7f53b8e: Status 404 returned error can't find the container with id 82c97d356253d27b77944c260eca3a47200509c96ebac45e65313c1ec7f53b8e
Sep 10 23:33:48.155143 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:48.155110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk" event={"ID":"08dca5fd-c227-41f6-aba8-48c5b9a2342e","Type":"ContainerStarted","Data":"93c5c49bc4331ac3731345b383df2c782b1f2cf477ffbc7f868d21e8d3ab7378"}
Sep 10 23:33:48.155565 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:48.155169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk" event={"ID":"08dca5fd-c227-41f6-aba8-48c5b9a2342e","Type":"ContainerStarted","Data":"82c97d356253d27b77944c260eca3a47200509c96ebac45e65313c1ec7f53b8e"}
Sep 10 23:33:48.169524 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:48.169363    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk" podStartSLOduration=1.169341425 podStartE2EDuration="1.169341425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:33:48.169004325 +0000 UTC m=+1948.860581530" watchObservedRunningTime="2026-09-10 23:33:48.169341425 +0000 UTC m=+1948.860918609"
Sep 10 23:33:49.162063 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:49.162029    2563 generic.go:356] "Generic (PLEG): container finished" podID="08dca5fd-c227-41f6-aba8-48c5b9a2342e" containerID="93c5c49bc4331ac3731345b383df2c782b1f2cf477ffbc7f868d21e8d3ab7378" exitCode=0
Sep 10 23:33:49.162441 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:49.162086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk" event={"ID":"08dca5fd-c227-41f6-aba8-48c5b9a2342e","Type":"ContainerDied","Data":"93c5c49bc4331ac3731345b383df2c782b1f2cf477ffbc7f868d21e8d3ab7378"}
Sep 10 23:33:50.295474 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:50.295449    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk"
Sep 10 23:33:50.395219 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:50.395181    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dca5fd-c227-41f6-aba8-48c5b9a2342e-maas-api-service-ca\") pod \"08dca5fd-c227-41f6-aba8-48c5b9a2342e\" (UID: \"08dca5fd-c227-41f6-aba8-48c5b9a2342e\") "
Sep 10 23:33:50.395518 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:50.395483    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08dca5fd-c227-41f6-aba8-48c5b9a2342e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08dca5fd-c227-41f6-aba8-48c5b9a2342e" (UID: "08dca5fd-c227-41f6-aba8-48c5b9a2342e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:33:50.496762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:50.496697    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dca5fd-c227-41f6-aba8-48c5b9a2342e-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:33:51.173486 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:51.173453    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk"
Sep 10 23:33:51.173651 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:51.173515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk" event={"ID":"08dca5fd-c227-41f6-aba8-48c5b9a2342e","Type":"ContainerDied","Data":"82c97d356253d27b77944c260eca3a47200509c96ebac45e65313c1ec7f53b8e"}
Sep 10 23:33:51.173651 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:51.173562    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82c97d356253d27b77944c260eca3a47200509c96ebac45e65313c1ec7f53b8e"
Sep 10 23:33:53.158572 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.158541    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"]
Sep 10 23:33:53.159022 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.158795    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm" podUID="9fd68b1c-8304-44fc-9110-b9bc149d17c9" containerName="payload-processing" containerID="cri-o://dfdcdbdd2a967f9b10e57a101ca418e9c0fe110aed049a1fd79639f8a6e22d2a" gracePeriod=30
Sep 10 23:33:53.165984 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.165953    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"]
Sep 10 23:33:53.166188 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.166163    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs" podUID="67c2460e-404f-40fe-9805-3bafc97baf77" containerName="payload-pre-processing" containerID="cri-o://cfb4219ec6a7604db93ce1ad85054585b0fe709e8ef28900c573d1a601c55007" gracePeriod=30
Sep 10 23:33:53.323277 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.323241    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk"]
Sep 10 23:33:53.326536 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.326485    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-230464-t27jk"]
Sep 10 23:33:53.448735 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.448710    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"
Sep 10 23:33:53.451668 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.451651    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"
Sep 10 23:33:53.624635 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.624603    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c2460e-404f-40fe-9805-3bafc97baf77-plugins-config-volume\") pod \"67c2460e-404f-40fe-9805-3bafc97baf77\" (UID: \"67c2460e-404f-40fe-9805-3bafc97baf77\") "
Sep 10 23:33:53.624829 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.624672    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcpfr\" (UniqueName: \"kubernetes.io/projected/9fd68b1c-8304-44fc-9110-b9bc149d17c9-kube-api-access-tcpfr\") pod \"9fd68b1c-8304-44fc-9110-b9bc149d17c9\" (UID: \"9fd68b1c-8304-44fc-9110-b9bc149d17c9\") "
Sep 10 23:33:53.624829 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.624742    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ds42\" (UniqueName: \"kubernetes.io/projected/67c2460e-404f-40fe-9805-3bafc97baf77-kube-api-access-5ds42\") pod \"67c2460e-404f-40fe-9805-3bafc97baf77\" (UID: \"67c2460e-404f-40fe-9805-3bafc97baf77\") "
Sep 10 23:33:53.624829 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.624783    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd68b1c-8304-44fc-9110-b9bc149d17c9-plugins-config-volume\") pod \"9fd68b1c-8304-44fc-9110-b9bc149d17c9\" (UID: \"9fd68b1c-8304-44fc-9110-b9bc149d17c9\") "
Sep 10 23:33:53.625058 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.625033    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c2460e-404f-40fe-9805-3bafc97baf77-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67c2460e-404f-40fe-9805-3bafc97baf77" (UID: "67c2460e-404f-40fe-9805-3bafc97baf77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:33:53.625256 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.625232    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd68b1c-8304-44fc-9110-b9bc149d17c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9fd68b1c-8304-44fc-9110-b9bc149d17c9" (UID: "9fd68b1c-8304-44fc-9110-b9bc149d17c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:33:53.626869 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.626845    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd68b1c-8304-44fc-9110-b9bc149d17c9-kube-api-access-tcpfr" (OuterVolumeSpecName: "kube-api-access-tcpfr") pod "9fd68b1c-8304-44fc-9110-b9bc149d17c9" (UID: "9fd68b1c-8304-44fc-9110-b9bc149d17c9"). InnerVolumeSpecName "kube-api-access-tcpfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:33:53.626869 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.626852    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c2460e-404f-40fe-9805-3bafc97baf77-kube-api-access-5ds42" (OuterVolumeSpecName: "kube-api-access-5ds42") pod "67c2460e-404f-40fe-9805-3bafc97baf77" (UID: "67c2460e-404f-40fe-9805-3bafc97baf77"). InnerVolumeSpecName "kube-api-access-5ds42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:33:53.726438 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.726378    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcpfr\" (UniqueName: \"kubernetes.io/projected/9fd68b1c-8304-44fc-9110-b9bc149d17c9-kube-api-access-tcpfr\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:33:53.726438 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.726407    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ds42\" (UniqueName: \"kubernetes.io/projected/67c2460e-404f-40fe-9805-3bafc97baf77-kube-api-access-5ds42\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:33:53.726438 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.726423    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd68b1c-8304-44fc-9110-b9bc149d17c9-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:33:53.726626 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.726437    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67c2460e-404f-40fe-9805-3bafc97baf77-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:33:53.815487 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:53.815459    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08dca5fd-c227-41f6-aba8-48c5b9a2342e" path="/var/lib/kubelet/pods/08dca5fd-c227-41f6-aba8-48c5b9a2342e/volumes"
Sep 10 23:33:54.191661 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.191624    2563 generic.go:356] "Generic (PLEG): container finished" podID="67c2460e-404f-40fe-9805-3bafc97baf77" containerID="cfb4219ec6a7604db93ce1ad85054585b0fe709e8ef28900c573d1a601c55007" exitCode=0
Sep 10 23:33:54.192056 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.191668    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"
Sep 10 23:33:54.192056 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.191703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs" event={"ID":"67c2460e-404f-40fe-9805-3bafc97baf77","Type":"ContainerDied","Data":"cfb4219ec6a7604db93ce1ad85054585b0fe709e8ef28900c573d1a601c55007"}
Sep 10 23:33:54.192056 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.191770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs" event={"ID":"67c2460e-404f-40fe-9805-3bafc97baf77","Type":"ContainerDied","Data":"1dcc73fd8817006daece30d42834381716f61787ca4e72df10012f3b83211ede"}
Sep 10 23:33:54.192056 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.191795    2563 scope.go:117] "RemoveContainer" containerID="cfb4219ec6a7604db93ce1ad85054585b0fe709e8ef28900c573d1a601c55007"
Sep 10 23:33:54.193346 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.193326    2563 generic.go:356] "Generic (PLEG): container finished" podID="9fd68b1c-8304-44fc-9110-b9bc149d17c9" containerID="dfdcdbdd2a967f9b10e57a101ca418e9c0fe110aed049a1fd79639f8a6e22d2a" exitCode=0
Sep 10 23:33:54.193455 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.193358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm" event={"ID":"9fd68b1c-8304-44fc-9110-b9bc149d17c9","Type":"ContainerDied","Data":"dfdcdbdd2a967f9b10e57a101ca418e9c0fe110aed049a1fd79639f8a6e22d2a"}
Sep 10 23:33:54.193455 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.193372    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"
Sep 10 23:33:54.193455 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.193410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm" event={"ID":"9fd68b1c-8304-44fc-9110-b9bc149d17c9","Type":"ContainerDied","Data":"1a8ff9077990f6ad0cac2b165fa7616ec888405532f56fd78a5cf6f03a1260a1"}
Sep 10 23:33:54.200890 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.200865    2563 scope.go:117] "RemoveContainer" containerID="cfb4219ec6a7604db93ce1ad85054585b0fe709e8ef28900c573d1a601c55007"
Sep 10 23:33:54.201155 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:33:54.201138    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb4219ec6a7604db93ce1ad85054585b0fe709e8ef28900c573d1a601c55007\": container with ID starting with cfb4219ec6a7604db93ce1ad85054585b0fe709e8ef28900c573d1a601c55007 not found: ID does not exist" containerID="cfb4219ec6a7604db93ce1ad85054585b0fe709e8ef28900c573d1a601c55007"
Sep 10 23:33:54.201201 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.201163    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb4219ec6a7604db93ce1ad85054585b0fe709e8ef28900c573d1a601c55007"} err="failed to get container status \"cfb4219ec6a7604db93ce1ad85054585b0fe709e8ef28900c573d1a601c55007\": rpc error: code = NotFound desc = could not find container \"cfb4219ec6a7604db93ce1ad85054585b0fe709e8ef28900c573d1a601c55007\": container with ID starting with cfb4219ec6a7604db93ce1ad85054585b0fe709e8ef28900c573d1a601c55007 not found: ID does not exist"
Sep 10 23:33:54.201201 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.201180    2563 scope.go:117] "RemoveContainer" containerID="dfdcdbdd2a967f9b10e57a101ca418e9c0fe110aed049a1fd79639f8a6e22d2a"
Sep 10 23:33:54.210505 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.210471    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"]
Sep 10 23:33:54.210744 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.210642    2563 scope.go:117] "RemoveContainer" containerID="dfdcdbdd2a967f9b10e57a101ca418e9c0fe110aed049a1fd79639f8a6e22d2a"
Sep 10 23:33:54.210919 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:33:54.210898    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfdcdbdd2a967f9b10e57a101ca418e9c0fe110aed049a1fd79639f8a6e22d2a\": container with ID starting with dfdcdbdd2a967f9b10e57a101ca418e9c0fe110aed049a1fd79639f8a6e22d2a not found: ID does not exist" containerID="dfdcdbdd2a967f9b10e57a101ca418e9c0fe110aed049a1fd79639f8a6e22d2a"
Sep 10 23:33:54.210980 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.210930    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfdcdbdd2a967f9b10e57a101ca418e9c0fe110aed049a1fd79639f8a6e22d2a"} err="failed to get container status \"dfdcdbdd2a967f9b10e57a101ca418e9c0fe110aed049a1fd79639f8a6e22d2a\": rpc error: code = NotFound desc = could not find container \"dfdcdbdd2a967f9b10e57a101ca418e9c0fe110aed049a1fd79639f8a6e22d2a\": container with ID starting with dfdcdbdd2a967f9b10e57a101ca418e9c0fe110aed049a1fd79639f8a6e22d2a not found: ID does not exist"
Sep 10 23:33:54.212801 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.212782    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-230464-79f9676c6b-c8xbs"]
Sep 10 23:33:54.219830 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.219811    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"]
Sep 10 23:33:54.223631 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:54.223612    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-230464-6b95d9cc49-dvvfm"]
Sep 10 23:33:55.814416 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:55.814371    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67c2460e-404f-40fe-9805-3bafc97baf77" path="/var/lib/kubelet/pods/67c2460e-404f-40fe-9805-3bafc97baf77/volumes"
Sep 10 23:33:55.815029 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:33:55.814950    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fd68b1c-8304-44fc-9110-b9bc149d17c9" path="/var/lib/kubelet/pods/9fd68b1c-8304-44fc-9110-b9bc149d17c9/volumes"
Sep 10 23:34:03.645060 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.645021    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4"]
Sep 10 23:34:03.645473 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.645460    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67c2460e-404f-40fe-9805-3bafc97baf77" containerName="payload-pre-processing"
Sep 10 23:34:03.645552 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.645475    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="67c2460e-404f-40fe-9805-3bafc97baf77" containerName="payload-pre-processing"
Sep 10 23:34:03.645552 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.645506    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08dca5fd-c227-41f6-aba8-48c5b9a2342e" containerName="revoke-keys"
Sep 10 23:34:03.645552 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.645513    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="08dca5fd-c227-41f6-aba8-48c5b9a2342e" containerName="revoke-keys"
Sep 10 23:34:03.645552 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.645527    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fd68b1c-8304-44fc-9110-b9bc149d17c9" containerName="payload-processing"
Sep 10 23:34:03.645552 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.645533    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fd68b1c-8304-44fc-9110-b9bc149d17c9" containerName="payload-processing"
Sep 10 23:34:03.645722 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.645587    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="67c2460e-404f-40fe-9805-3bafc97baf77" containerName="payload-pre-processing"
Sep 10 23:34:03.645722 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.645596    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fd68b1c-8304-44fc-9110-b9bc149d17c9" containerName="payload-processing"
Sep 10 23:34:03.645722 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.645602    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="08dca5fd-c227-41f6-aba8-48c5b9a2342e" containerName="revoke-keys"
Sep 10 23:34:03.650078 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.650058    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4"
Sep 10 23:34:03.652618 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.652590    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:34:03.653457 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.653424    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:34:03.655809 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.655787    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4"]
Sep 10 23:34:03.717173 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.717135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98753d5c-a8c8-4d8c-8051-236e060c9027-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4\" (UID: \"98753d5c-a8c8-4d8c-8051-236e060c9027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4"
Sep 10 23:34:03.817830 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.817803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98753d5c-a8c8-4d8c-8051-236e060c9027-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4\" (UID: \"98753d5c-a8c8-4d8c-8051-236e060c9027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4"
Sep 10 23:34:03.818407 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.818385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98753d5c-a8c8-4d8c-8051-236e060c9027-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4\" (UID: \"98753d5c-a8c8-4d8c-8051-236e060c9027\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4"
Sep 10 23:34:03.962883 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:03.962795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4"
Sep 10 23:34:04.233439 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:04.233361    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4"]
Sep 10 23:34:04.236546 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:34:04.236519    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98753d5c_a8c8_4d8c_8051_236e060c9027.slice/crio-0b3a63634643af5ef4528eef81bb86b5b2e1c34f5afdee031d965a31235335da WatchSource:0}: Error finding container 0b3a63634643af5ef4528eef81bb86b5b2e1c34f5afdee031d965a31235335da: Status 404 returned error can't find the container with id 0b3a63634643af5ef4528eef81bb86b5b2e1c34f5afdee031d965a31235335da
Sep 10 23:34:04.251047 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:04.251023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4" event={"ID":"98753d5c-a8c8-4d8c-8051-236e060c9027","Type":"ContainerStarted","Data":"0b3a63634643af5ef4528eef81bb86b5b2e1c34f5afdee031d965a31235335da"}
Sep 10 23:34:05.257669 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:05.257568    2563 generic.go:356] "Generic (PLEG): container finished" podID="98753d5c-a8c8-4d8c-8051-236e060c9027" containerID="badd65d6df9884167ad3412d1a20f73887e7b27459ab12e69bfb24d25d6c1a08" exitCode=0
Sep 10 23:34:05.258029 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:05.257660    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4" event={"ID":"98753d5c-a8c8-4d8c-8051-236e060c9027","Type":"ContainerDied","Data":"badd65d6df9884167ad3412d1a20f73887e7b27459ab12e69bfb24d25d6c1a08"}
Sep 10 23:34:06.388746 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:06.388722    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4"
Sep 10 23:34:06.442392 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:06.442362    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98753d5c-a8c8-4d8c-8051-236e060c9027-maas-api-service-ca\") pod \"98753d5c-a8c8-4d8c-8051-236e060c9027\" (UID: \"98753d5c-a8c8-4d8c-8051-236e060c9027\") "
Sep 10 23:34:06.442754 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:06.442723    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98753d5c-a8c8-4d8c-8051-236e060c9027-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98753d5c-a8c8-4d8c-8051-236e060c9027" (UID: "98753d5c-a8c8-4d8c-8051-236e060c9027"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:34:06.542998 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:06.542931    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98753d5c-a8c8-4d8c-8051-236e060c9027-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:34:07.269724 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:07.269694    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4"
Sep 10 23:34:07.269724 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:07.269730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4" event={"ID":"98753d5c-a8c8-4d8c-8051-236e060c9027","Type":"ContainerDied","Data":"0b3a63634643af5ef4528eef81bb86b5b2e1c34f5afdee031d965a31235335da"}
Sep 10 23:34:07.269944 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:07.269778    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b3a63634643af5ef4528eef81bb86b5b2e1c34f5afdee031d965a31235335da"
Sep 10 23:34:19.279376 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:19.279290    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4"]
Sep 10 23:34:19.281310 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:19.281285    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ea2b31-th8k4"]
Sep 10 23:34:19.813977 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:19.813946    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98753d5c-a8c8-4d8c-8051-236e060c9027" path="/var/lib/kubelet/pods/98753d5c-a8c8-4d8c-8051-236e060c9027/volumes"
Sep 10 23:34:20.350205 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:20.350172    2563 scope.go:117] "RemoveContainer" containerID="ccefcac3f0518569822b0aa9d68bb2d12016f16b1740d8200c2c675cf97b31fa"
Sep 10 23:34:20.359585 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:20.359567    2563 scope.go:117] "RemoveContainer" containerID="2e49c0e8bad8125d9d2d901963fa42534e5ca832830d2eb99885383028763d2a"
Sep 10 23:34:24.635672 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:24.635636    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"]
Sep 10 23:34:39.654825 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.654767    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b" podUID="41537d81-4a7a-4585-9a6c-4b9d33d4af80" containerName="main" containerID="cri-o://eb5b6df0b8262614b70072e27619bd16832f83cdde1916cf2c264bce65393aa0" gracePeriod=45
Sep 10 23:34:39.908840 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.908775    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:34:39.951422 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.951386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-dshm\") pod \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") "
Sep 10 23:34:39.951654 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.951461    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41537d81-4a7a-4585-9a6c-4b9d33d4af80-tls-certs\") pod \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") "
Sep 10 23:34:39.951654 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.951520    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-tmp-dir\") pod \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") "
Sep 10 23:34:39.951654 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.951550    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjzlr\" (UniqueName: \"kubernetes.io/projected/41537d81-4a7a-4585-9a6c-4b9d33d4af80-kube-api-access-cjzlr\") pod \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") "
Sep 10 23:34:39.951840 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.951678    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-home\") pod \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") "
Sep 10 23:34:39.951840 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.951704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-model-cache\") pod \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\" (UID: \"41537d81-4a7a-4585-9a6c-4b9d33d4af80\") "
Sep 10 23:34:39.951840 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.951794    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41537d81-4a7a-4585-9a6c-4b9d33d4af80" (UID: "41537d81-4a7a-4585-9a6c-4b9d33d4af80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:34:39.952101 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.952037    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-home" (OuterVolumeSpecName: "home") pod "41537d81-4a7a-4585-9a6c-4b9d33d4af80" (UID: "41537d81-4a7a-4585-9a6c-4b9d33d4af80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:34:39.952101 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.952059    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:34:39.952101 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.952047    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-model-cache" (OuterVolumeSpecName: "model-cache") pod "41537d81-4a7a-4585-9a6c-4b9d33d4af80" (UID: "41537d81-4a7a-4585-9a6c-4b9d33d4af80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:34:39.953882 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.953859    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-dshm" (OuterVolumeSpecName: "dshm") pod "41537d81-4a7a-4585-9a6c-4b9d33d4af80" (UID: "41537d81-4a7a-4585-9a6c-4b9d33d4af80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:34:39.954054 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.954024    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41537d81-4a7a-4585-9a6c-4b9d33d4af80-kube-api-access-cjzlr" (OuterVolumeSpecName: "kube-api-access-cjzlr") pod "41537d81-4a7a-4585-9a6c-4b9d33d4af80" (UID: "41537d81-4a7a-4585-9a6c-4b9d33d4af80"). InnerVolumeSpecName "kube-api-access-cjzlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:34:39.954127 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:39.954053    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41537d81-4a7a-4585-9a6c-4b9d33d4af80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41537d81-4a7a-4585-9a6c-4b9d33d4af80" (UID: "41537d81-4a7a-4585-9a6c-4b9d33d4af80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:34:40.052790 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:40.052753    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjzlr\" (UniqueName: \"kubernetes.io/projected/41537d81-4a7a-4585-9a6c-4b9d33d4af80-kube-api-access-cjzlr\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:34:40.052790 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:40.052785    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:34:40.052790 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:40.052797    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:34:40.053032 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:40.052805    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41537d81-4a7a-4585-9a6c-4b9d33d4af80-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:34:40.053032 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:40.052814    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41537d81-4a7a-4585-9a6c-4b9d33d4af80-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:34:40.447429 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:40.447397    2563 generic.go:356] "Generic (PLEG): container finished" podID="41537d81-4a7a-4585-9a6c-4b9d33d4af80" containerID="eb5b6df0b8262614b70072e27619bd16832f83cdde1916cf2c264bce65393aa0" exitCode=0
Sep 10 23:34:40.447626 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:40.447440    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"
Sep 10 23:34:40.447626 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:40.447459    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b" event={"ID":"41537d81-4a7a-4585-9a6c-4b9d33d4af80","Type":"ContainerDied","Data":"eb5b6df0b8262614b70072e27619bd16832f83cdde1916cf2c264bce65393aa0"}
Sep 10 23:34:40.447626 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:40.447504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b" event={"ID":"41537d81-4a7a-4585-9a6c-4b9d33d4af80","Type":"ContainerDied","Data":"753ec7d63072dcbbbb126fe6fc629717afc2b3b92fc646af2100eda98fc6dbc5"}
Sep 10 23:34:40.447626 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:40.447516    2563 scope.go:117] "RemoveContainer" containerID="eb5b6df0b8262614b70072e27619bd16832f83cdde1916cf2c264bce65393aa0"
Sep 10 23:34:40.457297 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:40.457280    2563 scope.go:117] "RemoveContainer" containerID="eb5b6df0b8262614b70072e27619bd16832f83cdde1916cf2c264bce65393aa0"
Sep 10 23:34:40.457600 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:34:40.457580    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5b6df0b8262614b70072e27619bd16832f83cdde1916cf2c264bce65393aa0\": container with ID starting with eb5b6df0b8262614b70072e27619bd16832f83cdde1916cf2c264bce65393aa0 not found: ID does not exist" containerID="eb5b6df0b8262614b70072e27619bd16832f83cdde1916cf2c264bce65393aa0"
Sep 10 23:34:40.457678 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:40.457612    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5b6df0b8262614b70072e27619bd16832f83cdde1916cf2c264bce65393aa0"} err="failed to get container status \"eb5b6df0b8262614b70072e27619bd16832f83cdde1916cf2c264bce65393aa0\": rpc error: code = NotFound desc = could not find container \"eb5b6df0b8262614b70072e27619bd16832f83cdde1916cf2c264bce65393aa0\": container with ID starting with eb5b6df0b8262614b70072e27619bd16832f83cdde1916cf2c264bce65393aa0 not found: ID does not exist"
Sep 10 23:34:40.468125 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:40.468100    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"]
Sep 10 23:34:40.471521 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:40.471484    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-ea2b31/ipp-route-ea2b31-kserve-7474d5b58c-wvn7b"]
Sep 10 23:34:41.814041 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:34:41.814007    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41537d81-4a7a-4585-9a6c-4b9d33d4af80" path="/var/lib/kubelet/pods/41537d81-4a7a-4585-9a6c-4b9d33d4af80/volumes"
Sep 10 23:35:08.525514 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.525452    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp"]
Sep 10 23:35:08.525924 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.525901    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41537d81-4a7a-4585-9a6c-4b9d33d4af80" containerName="main"
Sep 10 23:35:08.525924 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.525914    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="41537d81-4a7a-4585-9a6c-4b9d33d4af80" containerName="main"
Sep 10 23:35:08.525997 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.525934    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98753d5c-a8c8-4d8c-8051-236e060c9027" containerName="revoke-keys"
Sep 10 23:35:08.525997 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.525940    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="98753d5c-a8c8-4d8c-8051-236e060c9027" containerName="revoke-keys"
Sep 10 23:35:08.526065 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.526007    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="98753d5c-a8c8-4d8c-8051-236e060c9027" containerName="revoke-keys"
Sep 10 23:35:08.526065 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.526016    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="41537d81-4a7a-4585-9a6c-4b9d33d4af80" containerName="main"
Sep 10 23:35:08.529140 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.529124    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp"
Sep 10 23:35:08.531429 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.531409    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:35:08.531602 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.531582    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:35:08.537119 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.537096    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp"]
Sep 10 23:35:08.613965 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.613931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ad53ef-1693-4c0b-8e4d-fd170e4edfbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp\" (UID: \"12ad53ef-1693-4c0b-8e4d-fd170e4edfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp"
Sep 10 23:35:08.715612 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.715572    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ad53ef-1693-4c0b-8e4d-fd170e4edfbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp\" (UID: \"12ad53ef-1693-4c0b-8e4d-fd170e4edfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp"
Sep 10 23:35:08.716255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.716230    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ad53ef-1693-4c0b-8e4d-fd170e4edfbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp\" (UID: \"12ad53ef-1693-4c0b-8e4d-fd170e4edfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp"
Sep 10 23:35:08.840758 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:08.840723    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp"
Sep 10 23:35:09.110698 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:09.110670    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp"]
Sep 10 23:35:09.112677 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:35:09.112648    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ad53ef_1693_4c0b_8e4d_fd170e4edfbe.slice/crio-9809c5ff1ceecf7c23dc44b4a2fb5939f8eb8981479b49f1b2130dd3b7f66666 WatchSource:0}: Error finding container 9809c5ff1ceecf7c23dc44b4a2fb5939f8eb8981479b49f1b2130dd3b7f66666: Status 404 returned error can't find the container with id 9809c5ff1ceecf7c23dc44b4a2fb5939f8eb8981479b49f1b2130dd3b7f66666
Sep 10 23:35:09.598429 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:09.598387    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp" event={"ID":"12ad53ef-1693-4c0b-8e4d-fd170e4edfbe","Type":"ContainerStarted","Data":"c549c4a8d59c06d2e0f6af1eba7349cc732fe01b7d941e51ff579574fb4fed38"}
Sep 10 23:35:09.598887 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:09.598445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp" event={"ID":"12ad53ef-1693-4c0b-8e4d-fd170e4edfbe","Type":"ContainerStarted","Data":"9809c5ff1ceecf7c23dc44b4a2fb5939f8eb8981479b49f1b2130dd3b7f66666"}
Sep 10 23:35:09.613465 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:09.613411    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp" podStartSLOduration=1.613395339 podStartE2EDuration="1.613395339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:35:09.612057384 +0000 UTC m=+2030.303634609" watchObservedRunningTime="2026-09-10 23:35:09.613395339 +0000 UTC m=+2030.304972564"
Sep 10 23:35:10.605320 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:10.605278    2563 generic.go:356] "Generic (PLEG): container finished" podID="12ad53ef-1693-4c0b-8e4d-fd170e4edfbe" containerID="c549c4a8d59c06d2e0f6af1eba7349cc732fe01b7d941e51ff579574fb4fed38" exitCode=0
Sep 10 23:35:10.605729 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:10.605367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp" event={"ID":"12ad53ef-1693-4c0b-8e4d-fd170e4edfbe","Type":"ContainerDied","Data":"c549c4a8d59c06d2e0f6af1eba7349cc732fe01b7d941e51ff579574fb4fed38"}
Sep 10 23:35:11.749052 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:11.749017    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp"
Sep 10 23:35:11.849139 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:11.849104    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ad53ef-1693-4c0b-8e4d-fd170e4edfbe-maas-api-service-ca\") pod \"12ad53ef-1693-4c0b-8e4d-fd170e4edfbe\" (UID: \"12ad53ef-1693-4c0b-8e4d-fd170e4edfbe\") "
Sep 10 23:35:11.849552 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:11.849530    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ad53ef-1693-4c0b-8e4d-fd170e4edfbe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12ad53ef-1693-4c0b-8e4d-fd170e4edfbe" (UID: "12ad53ef-1693-4c0b-8e4d-fd170e4edfbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:11.950684 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:11.950599    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12ad53ef-1693-4c0b-8e4d-fd170e4edfbe-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:12.617589 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:12.617555    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp" event={"ID":"12ad53ef-1693-4c0b-8e4d-fd170e4edfbe","Type":"ContainerDied","Data":"9809c5ff1ceecf7c23dc44b4a2fb5939f8eb8981479b49f1b2130dd3b7f66666"}
Sep 10 23:35:12.617589 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:12.617584    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp"
Sep 10 23:35:12.617803 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:12.617601    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9809c5ff1ceecf7c23dc44b4a2fb5939f8eb8981479b49f1b2130dd3b7f66666"
Sep 10 23:35:19.083025 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.082991    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"]
Sep 10 23:35:19.083428 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.083193    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2" podUID="711d62f2-2ef7-4137-a1ea-a9234bca6b02" containerName="payload-processing" containerID="cri-o://d7368e6c5851b87990ad745dcbf3a72c129cc53ccb4068a5664ac009eb3403b5" gracePeriod=30
Sep 10 23:35:19.090613 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.090580    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"]
Sep 10 23:35:19.090842 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.090816    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c" podUID="74abccaf-3996-427d-9e3d-04d0c9480645" containerName="payload-pre-processing" containerID="cri-o://e28c1e2a46c12c20f174fa8247b9be7fa95025aa7514eea585f5b95a3f013e9b" gracePeriod=30
Sep 10 23:35:19.287769 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.287715    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp"]
Sep 10 23:35:19.289779 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.289751    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-706b9d-vmhsp"]
Sep 10 23:35:19.341543 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.341467    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"
Sep 10 23:35:19.370466 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.370438    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"
Sep 10 23:35:19.521824 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.521789    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74abccaf-3996-427d-9e3d-04d0c9480645-plugins-config-volume\") pod \"74abccaf-3996-427d-9e3d-04d0c9480645\" (UID: \"74abccaf-3996-427d-9e3d-04d0c9480645\") "
Sep 10 23:35:19.522001 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.521873    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rcwd\" (UniqueName: \"kubernetes.io/projected/711d62f2-2ef7-4137-a1ea-a9234bca6b02-kube-api-access-5rcwd\") pod \"711d62f2-2ef7-4137-a1ea-a9234bca6b02\" (UID: \"711d62f2-2ef7-4137-a1ea-a9234bca6b02\") "
Sep 10 23:35:19.522001 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.521923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711d62f2-2ef7-4137-a1ea-a9234bca6b02-plugins-config-volume\") pod \"711d62f2-2ef7-4137-a1ea-a9234bca6b02\" (UID: \"711d62f2-2ef7-4137-a1ea-a9234bca6b02\") "
Sep 10 23:35:19.522001 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.521962    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92v22\" (UniqueName: \"kubernetes.io/projected/74abccaf-3996-427d-9e3d-04d0c9480645-kube-api-access-92v22\") pod \"74abccaf-3996-427d-9e3d-04d0c9480645\" (UID: \"74abccaf-3996-427d-9e3d-04d0c9480645\") "
Sep 10 23:35:19.522303 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.522276    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74abccaf-3996-427d-9e3d-04d0c9480645-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74abccaf-3996-427d-9e3d-04d0c9480645" (UID: "74abccaf-3996-427d-9e3d-04d0c9480645"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:19.522375 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.522301    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711d62f2-2ef7-4137-a1ea-a9234bca6b02-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "711d62f2-2ef7-4137-a1ea-a9234bca6b02" (UID: "711d62f2-2ef7-4137-a1ea-a9234bca6b02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:19.523997 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.523977    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711d62f2-2ef7-4137-a1ea-a9234bca6b02-kube-api-access-5rcwd" (OuterVolumeSpecName: "kube-api-access-5rcwd") pod "711d62f2-2ef7-4137-a1ea-a9234bca6b02" (UID: "711d62f2-2ef7-4137-a1ea-a9234bca6b02"). InnerVolumeSpecName "kube-api-access-5rcwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:19.524066 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.524002    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74abccaf-3996-427d-9e3d-04d0c9480645-kube-api-access-92v22" (OuterVolumeSpecName: "kube-api-access-92v22") pod "74abccaf-3996-427d-9e3d-04d0c9480645" (UID: "74abccaf-3996-427d-9e3d-04d0c9480645"). InnerVolumeSpecName "kube-api-access-92v22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:19.622812 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.622726    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74abccaf-3996-427d-9e3d-04d0c9480645-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:19.622812 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.622758    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rcwd\" (UniqueName: \"kubernetes.io/projected/711d62f2-2ef7-4137-a1ea-a9234bca6b02-kube-api-access-5rcwd\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:19.622812 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.622768    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711d62f2-2ef7-4137-a1ea-a9234bca6b02-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:19.622812 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.622777    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92v22\" (UniqueName: \"kubernetes.io/projected/74abccaf-3996-427d-9e3d-04d0c9480645-kube-api-access-92v22\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:19.654667 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.654624    2563 generic.go:356] "Generic (PLEG): container finished" podID="711d62f2-2ef7-4137-a1ea-a9234bca6b02" containerID="d7368e6c5851b87990ad745dcbf3a72c129cc53ccb4068a5664ac009eb3403b5" exitCode=0
Sep 10 23:35:19.654667 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.654669    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"
Sep 10 23:35:19.654914 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.654709    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2" event={"ID":"711d62f2-2ef7-4137-a1ea-a9234bca6b02","Type":"ContainerDied","Data":"d7368e6c5851b87990ad745dcbf3a72c129cc53ccb4068a5664ac009eb3403b5"}
Sep 10 23:35:19.654914 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.654760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2" event={"ID":"711d62f2-2ef7-4137-a1ea-a9234bca6b02","Type":"ContainerDied","Data":"10add6bb42d508fc86659393d95695dc35c04101cbeeb47770f75caea017e01e"}
Sep 10 23:35:19.654914 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.654787    2563 scope.go:117] "RemoveContainer" containerID="d7368e6c5851b87990ad745dcbf3a72c129cc53ccb4068a5664ac009eb3403b5"
Sep 10 23:35:19.656050 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.656025    2563 generic.go:356] "Generic (PLEG): container finished" podID="74abccaf-3996-427d-9e3d-04d0c9480645" containerID="e28c1e2a46c12c20f174fa8247b9be7fa95025aa7514eea585f5b95a3f013e9b" exitCode=0
Sep 10 23:35:19.656142 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.656065    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"
Sep 10 23:35:19.656142 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.656104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c" event={"ID":"74abccaf-3996-427d-9e3d-04d0c9480645","Type":"ContainerDied","Data":"e28c1e2a46c12c20f174fa8247b9be7fa95025aa7514eea585f5b95a3f013e9b"}
Sep 10 23:35:19.656263 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.656142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c" event={"ID":"74abccaf-3996-427d-9e3d-04d0c9480645","Type":"ContainerDied","Data":"68546c35fd5f13d31cb70677e428728d8692f1723650e2bfa6de0a0a8a336bda"}
Sep 10 23:35:19.664795 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.664776    2563 scope.go:117] "RemoveContainer" containerID="d7368e6c5851b87990ad745dcbf3a72c129cc53ccb4068a5664ac009eb3403b5"
Sep 10 23:35:19.665090 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:35:19.665071    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7368e6c5851b87990ad745dcbf3a72c129cc53ccb4068a5664ac009eb3403b5\": container with ID starting with d7368e6c5851b87990ad745dcbf3a72c129cc53ccb4068a5664ac009eb3403b5 not found: ID does not exist" containerID="d7368e6c5851b87990ad745dcbf3a72c129cc53ccb4068a5664ac009eb3403b5"
Sep 10 23:35:19.665171 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.665103    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7368e6c5851b87990ad745dcbf3a72c129cc53ccb4068a5664ac009eb3403b5"} err="failed to get container status \"d7368e6c5851b87990ad745dcbf3a72c129cc53ccb4068a5664ac009eb3403b5\": rpc error: code = NotFound desc = could not find container \"d7368e6c5851b87990ad745dcbf3a72c129cc53ccb4068a5664ac009eb3403b5\": container with ID starting with d7368e6c5851b87990ad745dcbf3a72c129cc53ccb4068a5664ac009eb3403b5 not found: ID does not exist"
Sep 10 23:35:19.665171 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.665128    2563 scope.go:117] "RemoveContainer" containerID="e28c1e2a46c12c20f174fa8247b9be7fa95025aa7514eea585f5b95a3f013e9b"
Sep 10 23:35:19.673287 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.673230    2563 scope.go:117] "RemoveContainer" containerID="e28c1e2a46c12c20f174fa8247b9be7fa95025aa7514eea585f5b95a3f013e9b"
Sep 10 23:35:19.673815 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:35:19.673785    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28c1e2a46c12c20f174fa8247b9be7fa95025aa7514eea585f5b95a3f013e9b\": container with ID starting with e28c1e2a46c12c20f174fa8247b9be7fa95025aa7514eea585f5b95a3f013e9b not found: ID does not exist" containerID="e28c1e2a46c12c20f174fa8247b9be7fa95025aa7514eea585f5b95a3f013e9b"
Sep 10 23:35:19.673942 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.673847    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28c1e2a46c12c20f174fa8247b9be7fa95025aa7514eea585f5b95a3f013e9b"} err="failed to get container status \"e28c1e2a46c12c20f174fa8247b9be7fa95025aa7514eea585f5b95a3f013e9b\": rpc error: code = NotFound desc = could not find container \"e28c1e2a46c12c20f174fa8247b9be7fa95025aa7514eea585f5b95a3f013e9b\": container with ID starting with e28c1e2a46c12c20f174fa8247b9be7fa95025aa7514eea585f5b95a3f013e9b not found: ID does not exist"
Sep 10 23:35:19.676044 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.676019    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"]
Sep 10 23:35:19.679366 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.679346    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-706b9d-ffd4595dc-zhcx2"]
Sep 10 23:35:19.688753 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.688732    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"]
Sep 10 23:35:19.693929 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.693905    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-706b9d-6cffdd7dd6-m5t2c"]
Sep 10 23:35:19.814883 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.814847    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12ad53ef-1693-4c0b-8e4d-fd170e4edfbe" path="/var/lib/kubelet/pods/12ad53ef-1693-4c0b-8e4d-fd170e4edfbe/volumes"
Sep 10 23:35:19.815217 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.815204    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="711d62f2-2ef7-4137-a1ea-a9234bca6b02" path="/var/lib/kubelet/pods/711d62f2-2ef7-4137-a1ea-a9234bca6b02/volumes"
Sep 10 23:35:19.815564 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:19.815551    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74abccaf-3996-427d-9e3d-04d0c9480645" path="/var/lib/kubelet/pods/74abccaf-3996-427d-9e3d-04d0c9480645/volumes"
Sep 10 23:35:20.390240 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:20.390146    2563 scope.go:117] "RemoveContainer" containerID="1c337b5eb128e5ade6f73abba489d79f666736455990fd81f31c58c16bed4c6c"
Sep 10 23:35:26.120455 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.120424    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr"]
Sep 10 23:35:26.121025 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.121005    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="711d62f2-2ef7-4137-a1ea-a9234bca6b02" containerName="payload-processing"
Sep 10 23:35:26.121113 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.121028    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="711d62f2-2ef7-4137-a1ea-a9234bca6b02" containerName="payload-processing"
Sep 10 23:35:26.121113 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.121066    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12ad53ef-1693-4c0b-8e4d-fd170e4edfbe" containerName="revoke-keys"
Sep 10 23:35:26.121113 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.121076    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="12ad53ef-1693-4c0b-8e4d-fd170e4edfbe" containerName="revoke-keys"
Sep 10 23:35:26.121113 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.121100    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74abccaf-3996-427d-9e3d-04d0c9480645" containerName="payload-pre-processing"
Sep 10 23:35:26.121113 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.121108    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="74abccaf-3996-427d-9e3d-04d0c9480645" containerName="payload-pre-processing"
Sep 10 23:35:26.121349 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.121206    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="74abccaf-3996-427d-9e3d-04d0c9480645" containerName="payload-pre-processing"
Sep 10 23:35:26.121349 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.121221    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="12ad53ef-1693-4c0b-8e4d-fd170e4edfbe" containerName="revoke-keys"
Sep 10 23:35:26.121349 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.121234    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="711d62f2-2ef7-4137-a1ea-a9234bca6b02" containerName="payload-processing"
Sep 10 23:35:26.124470 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.124451    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr"
Sep 10 23:35:26.126953 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.126929    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:35:26.127065 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.127001    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:35:26.133390 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.133361    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr"]
Sep 10 23:35:26.184469 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.184435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4cf592-5787-4f47-93ee-320cad68ecb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr\" (UID: \"3e4cf592-5787-4f47-93ee-320cad68ecb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr"
Sep 10 23:35:26.285262 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.285222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4cf592-5787-4f47-93ee-320cad68ecb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr\" (UID: \"3e4cf592-5787-4f47-93ee-320cad68ecb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr"
Sep 10 23:35:26.285895 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.285874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4cf592-5787-4f47-93ee-320cad68ecb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr\" (UID: \"3e4cf592-5787-4f47-93ee-320cad68ecb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr"
Sep 10 23:35:26.435910 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.435819    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr"
Sep 10 23:35:26.706762 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:26.706416    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr"]
Sep 10 23:35:26.709083 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:35:26.709038    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4cf592_5787_4f47_93ee_320cad68ecb8.slice/crio-a21c29f524d782b0e01570ebab4e13be9e0400968b93e8fdda49385a3163f92f WatchSource:0}: Error finding container a21c29f524d782b0e01570ebab4e13be9e0400968b93e8fdda49385a3163f92f: Status 404 returned error can't find the container with id a21c29f524d782b0e01570ebab4e13be9e0400968b93e8fdda49385a3163f92f
Sep 10 23:35:27.699132 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:27.699049    2563 generic.go:356] "Generic (PLEG): container finished" podID="3e4cf592-5787-4f47-93ee-320cad68ecb8" containerID="225f9528e27f5165409e5610f6a24efcf22aa78a82ee72def84a2e57511c4922" exitCode=0
Sep 10 23:35:27.699481 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:27.699141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr" event={"ID":"3e4cf592-5787-4f47-93ee-320cad68ecb8","Type":"ContainerDied","Data":"225f9528e27f5165409e5610f6a24efcf22aa78a82ee72def84a2e57511c4922"}
Sep 10 23:35:27.699481 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:27.699232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr" event={"ID":"3e4cf592-5787-4f47-93ee-320cad68ecb8","Type":"ContainerStarted","Data":"a21c29f524d782b0e01570ebab4e13be9e0400968b93e8fdda49385a3163f92f"}
Sep 10 23:35:28.836928 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:28.836902    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr"
Sep 10 23:35:28.913822 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:28.913774    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4cf592-5787-4f47-93ee-320cad68ecb8-maas-api-service-ca\") pod \"3e4cf592-5787-4f47-93ee-320cad68ecb8\" (UID: \"3e4cf592-5787-4f47-93ee-320cad68ecb8\") "
Sep 10 23:35:28.914163 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:28.914137    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e4cf592-5787-4f47-93ee-320cad68ecb8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3e4cf592-5787-4f47-93ee-320cad68ecb8" (UID: "3e4cf592-5787-4f47-93ee-320cad68ecb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:29.015523 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:29.015398    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e4cf592-5787-4f47-93ee-320cad68ecb8-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:29.711797 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:29.711763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr" event={"ID":"3e4cf592-5787-4f47-93ee-320cad68ecb8","Type":"ContainerDied","Data":"a21c29f524d782b0e01570ebab4e13be9e0400968b93e8fdda49385a3163f92f"}
Sep 10 23:35:29.711797 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:29.711800    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr"
Sep 10 23:35:29.712028 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:29.711814    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a21c29f524d782b0e01570ebab4e13be9e0400968b93e8fdda49385a3163f92f"
Sep 10 23:35:36.759655 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:36.759613    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"]
Sep 10 23:35:36.760105 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:36.759909    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm" podUID="536c942a-7d7c-4a02-8f18-72279c461161" containerName="payload-processing" containerID="cri-o://c3a32612e964ce40cbc149443798ab10cd84abb8317a186b9f8dd13f7066a736" gracePeriod=30
Sep 10 23:35:36.764952 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:36.764921    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"]
Sep 10 23:35:36.765372 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:36.765343    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn" podUID="a47a9031-6ac1-4b11-b16c-15be5e73639f" containerName="payload-pre-processing" containerID="cri-o://a41bae18700e9cb6fb7af8092ef069a44c429556ccd680ba09e7656d3b39ee72" gracePeriod=30
Sep 10 23:35:36.913544 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:36.913477    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr"]
Sep 10 23:35:36.918244 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:36.918205    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6c97f2-vxzqr"]
Sep 10 23:35:37.011123 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.011060    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"
Sep 10 23:35:37.048095 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.048063    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"
Sep 10 23:35:37.092749 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.092719    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536c942a-7d7c-4a02-8f18-72279c461161-plugins-config-volume\") pod \"536c942a-7d7c-4a02-8f18-72279c461161\" (UID: \"536c942a-7d7c-4a02-8f18-72279c461161\") "
Sep 10 23:35:37.092943 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.092834    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5smc\" (UniqueName: \"kubernetes.io/projected/536c942a-7d7c-4a02-8f18-72279c461161-kube-api-access-b5smc\") pod \"536c942a-7d7c-4a02-8f18-72279c461161\" (UID: \"536c942a-7d7c-4a02-8f18-72279c461161\") "
Sep 10 23:35:37.093132 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.093099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536c942a-7d7c-4a02-8f18-72279c461161-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "536c942a-7d7c-4a02-8f18-72279c461161" (UID: "536c942a-7d7c-4a02-8f18-72279c461161"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:37.094916 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.094887    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536c942a-7d7c-4a02-8f18-72279c461161-kube-api-access-b5smc" (OuterVolumeSpecName: "kube-api-access-b5smc") pod "536c942a-7d7c-4a02-8f18-72279c461161" (UID: "536c942a-7d7c-4a02-8f18-72279c461161"). InnerVolumeSpecName "kube-api-access-b5smc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:37.193677 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.193628    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47a9031-6ac1-4b11-b16c-15be5e73639f-plugins-config-volume\") pod \"a47a9031-6ac1-4b11-b16c-15be5e73639f\" (UID: \"a47a9031-6ac1-4b11-b16c-15be5e73639f\") "
Sep 10 23:35:37.193837 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.193811    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hlrg\" (UniqueName: \"kubernetes.io/projected/a47a9031-6ac1-4b11-b16c-15be5e73639f-kube-api-access-8hlrg\") pod \"a47a9031-6ac1-4b11-b16c-15be5e73639f\" (UID: \"a47a9031-6ac1-4b11-b16c-15be5e73639f\") "
Sep 10 23:35:37.193981 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.193969    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5smc\" (UniqueName: \"kubernetes.io/projected/536c942a-7d7c-4a02-8f18-72279c461161-kube-api-access-b5smc\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:37.194022 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.193985    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536c942a-7d7c-4a02-8f18-72279c461161-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:37.194067 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.194038    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47a9031-6ac1-4b11-b16c-15be5e73639f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a47a9031-6ac1-4b11-b16c-15be5e73639f" (UID: "a47a9031-6ac1-4b11-b16c-15be5e73639f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:37.195858 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.195826    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47a9031-6ac1-4b11-b16c-15be5e73639f-kube-api-access-8hlrg" (OuterVolumeSpecName: "kube-api-access-8hlrg") pod "a47a9031-6ac1-4b11-b16c-15be5e73639f" (UID: "a47a9031-6ac1-4b11-b16c-15be5e73639f"). InnerVolumeSpecName "kube-api-access-8hlrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:37.295098 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.295001    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hlrg\" (UniqueName: \"kubernetes.io/projected/a47a9031-6ac1-4b11-b16c-15be5e73639f-kube-api-access-8hlrg\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:37.295098 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.295043    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a47a9031-6ac1-4b11-b16c-15be5e73639f-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:37.760793 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.760756    2563 generic.go:356] "Generic (PLEG): container finished" podID="536c942a-7d7c-4a02-8f18-72279c461161" containerID="c3a32612e964ce40cbc149443798ab10cd84abb8317a186b9f8dd13f7066a736" exitCode=0
Sep 10 23:35:37.761255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.760804    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"
Sep 10 23:35:37.761255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.760843    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm" event={"ID":"536c942a-7d7c-4a02-8f18-72279c461161","Type":"ContainerDied","Data":"c3a32612e964ce40cbc149443798ab10cd84abb8317a186b9f8dd13f7066a736"}
Sep 10 23:35:37.761255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.760913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm" event={"ID":"536c942a-7d7c-4a02-8f18-72279c461161","Type":"ContainerDied","Data":"e684500c39b14fcb8ae0fbe7e9a26a5cb06ed95b582b27d9311acad8063c688c"}
Sep 10 23:35:37.761255 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.760949    2563 scope.go:117] "RemoveContainer" containerID="c3a32612e964ce40cbc149443798ab10cd84abb8317a186b9f8dd13f7066a736"
Sep 10 23:35:37.762455 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.762222    2563 generic.go:356] "Generic (PLEG): container finished" podID="a47a9031-6ac1-4b11-b16c-15be5e73639f" containerID="a41bae18700e9cb6fb7af8092ef069a44c429556ccd680ba09e7656d3b39ee72" exitCode=0
Sep 10 23:35:37.762455 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.762269    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"
Sep 10 23:35:37.762455 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.762288    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn" event={"ID":"a47a9031-6ac1-4b11-b16c-15be5e73639f","Type":"ContainerDied","Data":"a41bae18700e9cb6fb7af8092ef069a44c429556ccd680ba09e7656d3b39ee72"}
Sep 10 23:35:37.762455 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.762325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn" event={"ID":"a47a9031-6ac1-4b11-b16c-15be5e73639f","Type":"ContainerDied","Data":"eec8af806010cd155e25db0cbf55f260df95523eb2cfa1082551a2f0db36f033"}
Sep 10 23:35:37.770132 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.769862    2563 scope.go:117] "RemoveContainer" containerID="c3a32612e964ce40cbc149443798ab10cd84abb8317a186b9f8dd13f7066a736"
Sep 10 23:35:37.770300 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:35:37.770191    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a32612e964ce40cbc149443798ab10cd84abb8317a186b9f8dd13f7066a736\": container with ID starting with c3a32612e964ce40cbc149443798ab10cd84abb8317a186b9f8dd13f7066a736 not found: ID does not exist" containerID="c3a32612e964ce40cbc149443798ab10cd84abb8317a186b9f8dd13f7066a736"
Sep 10 23:35:37.770300 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.770222    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a32612e964ce40cbc149443798ab10cd84abb8317a186b9f8dd13f7066a736"} err="failed to get container status \"c3a32612e964ce40cbc149443798ab10cd84abb8317a186b9f8dd13f7066a736\": rpc error: code = NotFound desc = could not find container \"c3a32612e964ce40cbc149443798ab10cd84abb8317a186b9f8dd13f7066a736\": container with ID starting with c3a32612e964ce40cbc149443798ab10cd84abb8317a186b9f8dd13f7066a736 not found: ID does not exist"
Sep 10 23:35:37.770300 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.770244    2563 scope.go:117] "RemoveContainer" containerID="a41bae18700e9cb6fb7af8092ef069a44c429556ccd680ba09e7656d3b39ee72"
Sep 10 23:35:37.784004 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.783987    2563 scope.go:117] "RemoveContainer" containerID="a41bae18700e9cb6fb7af8092ef069a44c429556ccd680ba09e7656d3b39ee72"
Sep 10 23:35:37.784279 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:35:37.784262    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41bae18700e9cb6fb7af8092ef069a44c429556ccd680ba09e7656d3b39ee72\": container with ID starting with a41bae18700e9cb6fb7af8092ef069a44c429556ccd680ba09e7656d3b39ee72 not found: ID does not exist" containerID="a41bae18700e9cb6fb7af8092ef069a44c429556ccd680ba09e7656d3b39ee72"
Sep 10 23:35:37.784319 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.784288    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41bae18700e9cb6fb7af8092ef069a44c429556ccd680ba09e7656d3b39ee72"} err="failed to get container status \"a41bae18700e9cb6fb7af8092ef069a44c429556ccd680ba09e7656d3b39ee72\": rpc error: code = NotFound desc = could not find container \"a41bae18700e9cb6fb7af8092ef069a44c429556ccd680ba09e7656d3b39ee72\": container with ID starting with a41bae18700e9cb6fb7af8092ef069a44c429556ccd680ba09e7656d3b39ee72 not found: ID does not exist"
Sep 10 23:35:37.792301 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.792274    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"]
Sep 10 23:35:37.795955 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.795933    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6c97f2-5879ccf68b-fhjkn"]
Sep 10 23:35:37.814354 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.814320    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e4cf592-5787-4f47-93ee-320cad68ecb8" path="/var/lib/kubelet/pods/3e4cf592-5787-4f47-93ee-320cad68ecb8/volumes"
Sep 10 23:35:37.814777 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.814756    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a47a9031-6ac1-4b11-b16c-15be5e73639f" path="/var/lib/kubelet/pods/a47a9031-6ac1-4b11-b16c-15be5e73639f/volumes"
Sep 10 23:35:37.815048 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.815031    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"]
Sep 10 23:35:37.815113 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:37.815063    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6c97f2-6b499fd7cc-gz6pm"]
Sep 10 23:35:39.815707 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:39.815673    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536c942a-7d7c-4a02-8f18-72279c461161" path="/var/lib/kubelet/pods/536c942a-7d7c-4a02-8f18-72279c461161/volumes"
Sep 10 23:35:43.868761 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.868681    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn"]
Sep 10 23:35:43.869133 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.869061    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e4cf592-5787-4f47-93ee-320cad68ecb8" containerName="revoke-keys"
Sep 10 23:35:43.869133 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.869073    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e4cf592-5787-4f47-93ee-320cad68ecb8" containerName="revoke-keys"
Sep 10 23:35:43.869133 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.869086    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536c942a-7d7c-4a02-8f18-72279c461161" containerName="payload-processing"
Sep 10 23:35:43.869133 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.869091    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="536c942a-7d7c-4a02-8f18-72279c461161" containerName="payload-processing"
Sep 10 23:35:43.869133 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.869104    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a47a9031-6ac1-4b11-b16c-15be5e73639f" containerName="payload-pre-processing"
Sep 10 23:35:43.869133 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.869110    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a47a9031-6ac1-4b11-b16c-15be5e73639f" containerName="payload-pre-processing"
Sep 10 23:35:43.869328 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.869159    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="536c942a-7d7c-4a02-8f18-72279c461161" containerName="payload-processing"
Sep 10 23:35:43.869328 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.869167    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e4cf592-5787-4f47-93ee-320cad68ecb8" containerName="revoke-keys"
Sep 10 23:35:43.869328 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.869174    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a47a9031-6ac1-4b11-b16c-15be5e73639f" containerName="payload-pre-processing"
Sep 10 23:35:43.873835 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.873816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn"
Sep 10 23:35:43.878136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.876719    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:35:43.878136 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.877176    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5jwx7\""
Sep 10 23:35:43.882560 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.882120    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn"]
Sep 10 23:35:43.954784 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:43.954749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd423617-eb2c-411b-b659-5b80c8ec4eec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn\" (UID: \"fd423617-eb2c-411b-b659-5b80c8ec4eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn"
Sep 10 23:35:44.055923 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:44.055887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd423617-eb2c-411b-b659-5b80c8ec4eec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn\" (UID: \"fd423617-eb2c-411b-b659-5b80c8ec4eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn"
Sep 10 23:35:44.056541 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:44.056522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd423617-eb2c-411b-b659-5b80c8ec4eec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn\" (UID: \"fd423617-eb2c-411b-b659-5b80c8ec4eec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn"
Sep 10 23:35:44.187448 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:44.187364    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn"
Sep 10 23:35:44.458615 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:44.458405    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn"]
Sep 10 23:35:44.461473 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:35:44.461448    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd423617_eb2c_411b_b659_5b80c8ec4eec.slice/crio-5d1a506b9d7d36330ec0e64bc7529cdf6b5530bc6af9492ee28d60ca334e2f01 WatchSource:0}: Error finding container 5d1a506b9d7d36330ec0e64bc7529cdf6b5530bc6af9492ee28d60ca334e2f01: Status 404 returned error can't find the container with id 5d1a506b9d7d36330ec0e64bc7529cdf6b5530bc6af9492ee28d60ca334e2f01
Sep 10 23:35:44.801061 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:44.800973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn" event={"ID":"fd423617-eb2c-411b-b659-5b80c8ec4eec","Type":"ContainerStarted","Data":"b08369da38944ae27951d2d90e2c3a4a65db174d5dc9453950c77e4d3aa2fdb1"}
Sep 10 23:35:44.801061 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:44.801032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn" event={"ID":"fd423617-eb2c-411b-b659-5b80c8ec4eec","Type":"ContainerStarted","Data":"5d1a506b9d7d36330ec0e64bc7529cdf6b5530bc6af9492ee28d60ca334e2f01"}
Sep 10 23:35:44.815053 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:44.815008    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn" podStartSLOduration=1.814993396 podStartE2EDuration="1.814993396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:35:44.814577934 +0000 UTC m=+2065.506155137" watchObservedRunningTime="2026-09-10 23:35:44.814993396 +0000 UTC m=+2065.506570579"
Sep 10 23:35:45.808038 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:45.808007    2563 generic.go:356] "Generic (PLEG): container finished" podID="fd423617-eb2c-411b-b659-5b80c8ec4eec" containerID="b08369da38944ae27951d2d90e2c3a4a65db174d5dc9453950c77e4d3aa2fdb1" exitCode=0
Sep 10 23:35:45.813217 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:45.813191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn" event={"ID":"fd423617-eb2c-411b-b659-5b80c8ec4eec","Type":"ContainerDied","Data":"b08369da38944ae27951d2d90e2c3a4a65db174d5dc9453950c77e4d3aa2fdb1"}
Sep 10 23:35:46.941847 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:46.941825    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn"
Sep 10 23:35:47.085226 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:47.085142    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd423617-eb2c-411b-b659-5b80c8ec4eec-maas-api-service-ca\") pod \"fd423617-eb2c-411b-b659-5b80c8ec4eec\" (UID: \"fd423617-eb2c-411b-b659-5b80c8ec4eec\") "
Sep 10 23:35:47.085550 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:47.085524    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd423617-eb2c-411b-b659-5b80c8ec4eec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd423617-eb2c-411b-b659-5b80c8ec4eec" (UID: "fd423617-eb2c-411b-b659-5b80c8ec4eec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:47.186039 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:47.186003    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd423617-eb2c-411b-b659-5b80c8ec4eec-maas-api-service-ca\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:47.819472 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:47.819442    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn"
Sep 10 23:35:47.819671 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:47.819440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn" event={"ID":"fd423617-eb2c-411b-b659-5b80c8ec4eec","Type":"ContainerDied","Data":"5d1a506b9d7d36330ec0e64bc7529cdf6b5530bc6af9492ee28d60ca334e2f01"}
Sep 10 23:35:47.819671 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:47.819583    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d1a506b9d7d36330ec0e64bc7529cdf6b5530bc6af9492ee28d60ca334e2f01"
Sep 10 23:35:54.309267 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.309230    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-83be66-29818050-w8sdd"]
Sep 10 23:35:54.312926 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.312900    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-83be66-29818050-w8sdd"]
Sep 10 23:35:54.336220 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.336188    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"]
Sep 10 23:35:54.336407 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.336380    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr" podUID="c238769d-e4f5-443a-aa41-26cfb97da683" containerName="payload-processing" containerID="cri-o://f8544377577d97683b60b4ca6fff3fe6625bec9e309f1dc9be6a8ac2137ea2ce" gracePeriod=30
Sep 10 23:35:54.343407 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.343373    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"]
Sep 10 23:35:54.343659 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.343632    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2" podUID="67636cdc-1284-4dc0-801c-4d90ee4e5c67" containerName="payload-pre-processing" containerID="cri-o://160427bbf34802a716999af583b63387da491735ab65b50fae21b9963afe4b25" gracePeriod=30
Sep 10 23:35:54.492282 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.492249    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn"]
Sep 10 23:35:54.494937 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.494914    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-83be66-j5tqn"]
Sep 10 23:35:54.609662 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.609634    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"
Sep 10 23:35:54.612319 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.612299    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"
Sep 10 23:35:54.755582 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.755546    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdnvd\" (UniqueName: \"kubernetes.io/projected/c238769d-e4f5-443a-aa41-26cfb97da683-kube-api-access-xdnvd\") pod \"c238769d-e4f5-443a-aa41-26cfb97da683\" (UID: \"c238769d-e4f5-443a-aa41-26cfb97da683\") "
Sep 10 23:35:54.755767 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.755622    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c238769d-e4f5-443a-aa41-26cfb97da683-plugins-config-volume\") pod \"c238769d-e4f5-443a-aa41-26cfb97da683\" (UID: \"c238769d-e4f5-443a-aa41-26cfb97da683\") "
Sep 10 23:35:54.755767 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.755651    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67636cdc-1284-4dc0-801c-4d90ee4e5c67-plugins-config-volume\") pod \"67636cdc-1284-4dc0-801c-4d90ee4e5c67\" (UID: \"67636cdc-1284-4dc0-801c-4d90ee4e5c67\") "
Sep 10 23:35:54.755767 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.755717    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pncgm\" (UniqueName: \"kubernetes.io/projected/67636cdc-1284-4dc0-801c-4d90ee4e5c67-kube-api-access-pncgm\") pod \"67636cdc-1284-4dc0-801c-4d90ee4e5c67\" (UID: \"67636cdc-1284-4dc0-801c-4d90ee4e5c67\") "
Sep 10 23:35:54.756041 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.756013    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c238769d-e4f5-443a-aa41-26cfb97da683-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c238769d-e4f5-443a-aa41-26cfb97da683" (UID: "c238769d-e4f5-443a-aa41-26cfb97da683"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:54.756112 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.756086    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67636cdc-1284-4dc0-801c-4d90ee4e5c67-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67636cdc-1284-4dc0-801c-4d90ee4e5c67" (UID: "67636cdc-1284-4dc0-801c-4d90ee4e5c67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:54.757764 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.757740    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67636cdc-1284-4dc0-801c-4d90ee4e5c67-kube-api-access-pncgm" (OuterVolumeSpecName: "kube-api-access-pncgm") pod "67636cdc-1284-4dc0-801c-4d90ee4e5c67" (UID: "67636cdc-1284-4dc0-801c-4d90ee4e5c67"). InnerVolumeSpecName "kube-api-access-pncgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:54.757875 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.757836    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c238769d-e4f5-443a-aa41-26cfb97da683-kube-api-access-xdnvd" (OuterVolumeSpecName: "kube-api-access-xdnvd") pod "c238769d-e4f5-443a-aa41-26cfb97da683" (UID: "c238769d-e4f5-443a-aa41-26cfb97da683"). InnerVolumeSpecName "kube-api-access-xdnvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:54.856910 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.856819    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pncgm\" (UniqueName: \"kubernetes.io/projected/67636cdc-1284-4dc0-801c-4d90ee4e5c67-kube-api-access-pncgm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:54.856910 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.856845    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdnvd\" (UniqueName: \"kubernetes.io/projected/c238769d-e4f5-443a-aa41-26cfb97da683-kube-api-access-xdnvd\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:54.856910 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.856855    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c238769d-e4f5-443a-aa41-26cfb97da683-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:54.856910 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.856864    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67636cdc-1284-4dc0-801c-4d90ee4e5c67-plugins-config-volume\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:35:54.861310 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.861282    2563 generic.go:356] "Generic (PLEG): container finished" podID="c238769d-e4f5-443a-aa41-26cfb97da683" containerID="f8544377577d97683b60b4ca6fff3fe6625bec9e309f1dc9be6a8ac2137ea2ce" exitCode=0
Sep 10 23:35:54.861459 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.861350    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"
Sep 10 23:35:54.861459 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.861363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr" event={"ID":"c238769d-e4f5-443a-aa41-26cfb97da683","Type":"ContainerDied","Data":"f8544377577d97683b60b4ca6fff3fe6625bec9e309f1dc9be6a8ac2137ea2ce"}
Sep 10 23:35:54.861459 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.861424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr" event={"ID":"c238769d-e4f5-443a-aa41-26cfb97da683","Type":"ContainerDied","Data":"f9aa34489aeb065abd7072e4f4a1480570ae7a489250ba418ff3120b951a0a38"}
Sep 10 23:35:54.861459 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.861453    2563 scope.go:117] "RemoveContainer" containerID="f8544377577d97683b60b4ca6fff3fe6625bec9e309f1dc9be6a8ac2137ea2ce"
Sep 10 23:35:54.862892 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.862867    2563 generic.go:356] "Generic (PLEG): container finished" podID="67636cdc-1284-4dc0-801c-4d90ee4e5c67" containerID="160427bbf34802a716999af583b63387da491735ab65b50fae21b9963afe4b25" exitCode=0
Sep 10 23:35:54.862987 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.862905    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"
Sep 10 23:35:54.862987 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.862909    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2" event={"ID":"67636cdc-1284-4dc0-801c-4d90ee4e5c67","Type":"ContainerDied","Data":"160427bbf34802a716999af583b63387da491735ab65b50fae21b9963afe4b25"}
Sep 10 23:35:54.863069 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.862983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2" event={"ID":"67636cdc-1284-4dc0-801c-4d90ee4e5c67","Type":"ContainerDied","Data":"47f1fe3a4cd19674555607586899594d97131442f571059c74034f7746dc2af7"}
Sep 10 23:35:54.871163 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.871143    2563 scope.go:117] "RemoveContainer" containerID="f8544377577d97683b60b4ca6fff3fe6625bec9e309f1dc9be6a8ac2137ea2ce"
Sep 10 23:35:54.871435 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:35:54.871417    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8544377577d97683b60b4ca6fff3fe6625bec9e309f1dc9be6a8ac2137ea2ce\": container with ID starting with f8544377577d97683b60b4ca6fff3fe6625bec9e309f1dc9be6a8ac2137ea2ce not found: ID does not exist" containerID="f8544377577d97683b60b4ca6fff3fe6625bec9e309f1dc9be6a8ac2137ea2ce"
Sep 10 23:35:54.871511 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.871444    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8544377577d97683b60b4ca6fff3fe6625bec9e309f1dc9be6a8ac2137ea2ce"} err="failed to get container status \"f8544377577d97683b60b4ca6fff3fe6625bec9e309f1dc9be6a8ac2137ea2ce\": rpc error: code = NotFound desc = could not find container \"f8544377577d97683b60b4ca6fff3fe6625bec9e309f1dc9be6a8ac2137ea2ce\": container with ID starting with f8544377577d97683b60b4ca6fff3fe6625bec9e309f1dc9be6a8ac2137ea2ce not found: ID does not exist"
Sep 10 23:35:54.871511 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.871464    2563 scope.go:117] "RemoveContainer" containerID="160427bbf34802a716999af583b63387da491735ab65b50fae21b9963afe4b25"
Sep 10 23:35:54.879916 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.879898    2563 scope.go:117] "RemoveContainer" containerID="160427bbf34802a716999af583b63387da491735ab65b50fae21b9963afe4b25"
Sep 10 23:35:54.880199 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:35:54.880180    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"160427bbf34802a716999af583b63387da491735ab65b50fae21b9963afe4b25\": container with ID starting with 160427bbf34802a716999af583b63387da491735ab65b50fae21b9963afe4b25 not found: ID does not exist" containerID="160427bbf34802a716999af583b63387da491735ab65b50fae21b9963afe4b25"
Sep 10 23:35:54.880271 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.880206    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160427bbf34802a716999af583b63387da491735ab65b50fae21b9963afe4b25"} err="failed to get container status \"160427bbf34802a716999af583b63387da491735ab65b50fae21b9963afe4b25\": rpc error: code = NotFound desc = could not find container \"160427bbf34802a716999af583b63387da491735ab65b50fae21b9963afe4b25\": container with ID starting with 160427bbf34802a716999af583b63387da491735ab65b50fae21b9963afe4b25 not found: ID does not exist"
Sep 10 23:35:54.885687 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.885663    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"]
Sep 10 23:35:54.891406 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.891381    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-83be66-744498577d-hz4r2"]
Sep 10 23:35:54.900949 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.900920    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"]
Sep 10 23:35:54.902484 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:54.902461    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-83be66-6769cb6845-pqvwr"]
Sep 10 23:35:55.814453 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:55.814416    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67636cdc-1284-4dc0-801c-4d90ee4e5c67" path="/var/lib/kubelet/pods/67636cdc-1284-4dc0-801c-4d90ee4e5c67/volumes"
Sep 10 23:35:55.814859 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:55.814742    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8819aba5-4867-4d59-bad2-b74f8c9c2ed2" path="/var/lib/kubelet/pods/8819aba5-4867-4d59-bad2-b74f8c9c2ed2/volumes"
Sep 10 23:35:55.815026 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:55.815013    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c238769d-e4f5-443a-aa41-26cfb97da683" path="/var/lib/kubelet/pods/c238769d-e4f5-443a-aa41-26cfb97da683/volumes"
Sep 10 23:35:55.815292 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:35:55.815281    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd423617-eb2c-411b-b659-5b80c8ec4eec" path="/var/lib/kubelet/pods/fd423617-eb2c-411b-b659-5b80c8ec4eec/volumes"
Sep 10 23:36:19.959478 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:36:19.959453    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:36:19.961006 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:36:19.960982    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:36:20.425584 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:36:20.425552    2563 scope.go:117] "RemoveContainer" containerID="3ec911f636e279833fb32029abef3abb3ec0c201f5a2d03974627eeea05e1880"
Sep 10 23:40:20.482226 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:20.482148    2563 scope.go:117] "RemoveContainer" containerID="badd65d6df9884167ad3412d1a20f73887e7b27459ab12e69bfb24d25d6c1a08"
Sep 10 23:40:20.490076 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:20.490057    2563 scope.go:117] "RemoveContainer" containerID="93c5c49bc4331ac3731345b383df2c782b1f2cf477ffbc7f868d21e8d3ab7378"
Sep 10 23:40:21.637328 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.637296    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"]
Sep 10 23:40:21.637768 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.637663    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c238769d-e4f5-443a-aa41-26cfb97da683" containerName="payload-processing"
Sep 10 23:40:21.637768 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.637677    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c238769d-e4f5-443a-aa41-26cfb97da683" containerName="payload-processing"
Sep 10 23:40:21.637768 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.637690    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd423617-eb2c-411b-b659-5b80c8ec4eec" containerName="revoke-keys"
Sep 10 23:40:21.637768 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.637696    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd423617-eb2c-411b-b659-5b80c8ec4eec" containerName="revoke-keys"
Sep 10 23:40:21.637768 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.637708    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67636cdc-1284-4dc0-801c-4d90ee4e5c67" containerName="payload-pre-processing"
Sep 10 23:40:21.637768 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.637713    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="67636cdc-1284-4dc0-801c-4d90ee4e5c67" containerName="payload-pre-processing"
Sep 10 23:40:21.638024 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.637812    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd423617-eb2c-411b-b659-5b80c8ec4eec" containerName="revoke-keys"
Sep 10 23:40:21.638024 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.637825    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c238769d-e4f5-443a-aa41-26cfb97da683" containerName="payload-processing"
Sep 10 23:40:21.638024 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.637836    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="67636cdc-1284-4dc0-801c-4d90ee4e5c67" containerName="payload-pre-processing"
Sep 10 23:40:21.642287 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.642268    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.644519 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.644483    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-7baf1b07-kserve-self-signed-certs\""
Sep 10 23:40:21.650242 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.650218    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"]
Sep 10 23:40:21.761352 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.761310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-home\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.761352 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.761354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-tmp-dir\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.761615 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.761386    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-dshm\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.761615 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.761436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-model-cache\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.761615 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.761481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3aaa50-238e-4b71-be38-73b557d391d3-tls-certs\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.761615 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.761565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vc6h\" (UniqueName: \"kubernetes.io/projected/3e3aaa50-238e-4b71-be38-73b557d391d3-kube-api-access-8vc6h\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.862229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.862193    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-home\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.862229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.862230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-tmp-dir\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.862530 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.862260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-dshm\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.862530 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.862299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-model-cache\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.862530 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.862318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3aaa50-238e-4b71-be38-73b557d391d3-tls-certs\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.862530 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.862349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vc6h\" (UniqueName: \"kubernetes.io/projected/3e3aaa50-238e-4b71-be38-73b557d391d3-kube-api-access-8vc6h\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.862753 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.862694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-home\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.862753 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.862729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-tmp-dir\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.862827 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.862736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-model-cache\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.864675 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.864642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-dshm\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.864904 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.864885    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3aaa50-238e-4b71-be38-73b557d391d3-tls-certs\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.869567 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.869545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vc6h\" (UniqueName: \"kubernetes.io/projected/3e3aaa50-238e-4b71-be38-73b557d391d3-kube-api-access-8vc6h\") pod \"e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") " pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:21.951874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:21.951800    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:22.224486 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:22.224403    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"]
Sep 10 23:40:22.228359 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:40:22.228328    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e3aaa50_238e_4b71_be38_73b557d391d3.slice/crio-d5db8ff5099331e703b3f320f0517b3203e13554a6dd99b1897c59bb392083c7 WatchSource:0}: Error finding container d5db8ff5099331e703b3f320f0517b3203e13554a6dd99b1897c59bb392083c7: Status 404 returned error can't find the container with id d5db8ff5099331e703b3f320f0517b3203e13554a6dd99b1897c59bb392083c7
Sep 10 23:40:23.199837 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:23.199783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687" event={"ID":"3e3aaa50-238e-4b71-be38-73b557d391d3","Type":"ContainerStarted","Data":"07b464bc9ac3cb3a7b7b4b8a2d363ab6541fdd1b6eb97a6fbe29fe8552d2dd04"}
Sep 10 23:40:23.199837 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:23.199838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687" event={"ID":"3e3aaa50-238e-4b71-be38-73b557d391d3","Type":"ContainerStarted","Data":"d5db8ff5099331e703b3f320f0517b3203e13554a6dd99b1897c59bb392083c7"}
Sep 10 23:40:23.216533 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:23.216471    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687" podStartSLOduration=2.216456307 podStartE2EDuration="2.216456307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:40:23.215480461 +0000 UTC m=+2343.907057655" watchObservedRunningTime="2026-09-10 23:40:23.216456307 +0000 UTC m=+2343.908033490"
Sep 10 23:40:25.260654 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.260613    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"]
Sep 10 23:40:25.264118 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.264091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.266427 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.266403    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-7baf1b07-kserve-self-signed-certs\""
Sep 10 23:40:25.273050 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.272579    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"]
Sep 10 23:40:25.397485 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.397436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-dshm\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.397701 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.397521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffbb9\" (UniqueName: \"kubernetes.io/projected/f1a55d86-e0a3-475a-bfca-9683afdea519-kube-api-access-ffbb9\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.397701 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.397567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1a55d86-e0a3-475a-bfca-9683afdea519-tls-certs\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.397701 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.397592    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-model-cache\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.397701 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.397622    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-tmp-dir\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.397701 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.397645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-home\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.498556 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.498515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-dshm\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.498768 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.498634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffbb9\" (UniqueName: \"kubernetes.io/projected/f1a55d86-e0a3-475a-bfca-9683afdea519-kube-api-access-ffbb9\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.498768 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.498669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1a55d86-e0a3-475a-bfca-9683afdea519-tls-certs\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.498768 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.498689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-model-cache\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.498768 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.498710    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-tmp-dir\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.498768 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.498726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-home\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.499096 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.499064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-model-cache\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.499175 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.499116    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-home\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.499175 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.499135    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-tmp-dir\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.500964 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.500938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-dshm\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.501112 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.501096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1a55d86-e0a3-475a-bfca-9683afdea519-tls-certs\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.506983 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.506962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffbb9\" (UniqueName: \"kubernetes.io/projected/f1a55d86-e0a3-475a-bfca-9683afdea519-kube-api-access-ffbb9\") pod \"e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") " pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.575393 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.575347    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:25.847182 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:25.847097    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"]
Sep 10 23:40:25.850339 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:40:25.850308    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1a55d86_e0a3_475a_bfca_9683afdea519.slice/crio-f63519959bd9e61df7c645b9ef003cd85e2e2df6fcca69f3cf60fabdc2f21095 WatchSource:0}: Error finding container f63519959bd9e61df7c645b9ef003cd85e2e2df6fcca69f3cf60fabdc2f21095: Status 404 returned error can't find the container with id f63519959bd9e61df7c645b9ef003cd85e2e2df6fcca69f3cf60fabdc2f21095
Sep 10 23:40:26.218306 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:26.218213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts" event={"ID":"f1a55d86-e0a3-475a-bfca-9683afdea519","Type":"ContainerStarted","Data":"60c0681b9f00c02b1252eac9a42ad0a3ce4fba6406cd409a3d74376eb1e7d8a2"}
Sep 10 23:40:26.218306 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:26.218257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts" event={"ID":"f1a55d86-e0a3-475a-bfca-9683afdea519","Type":"ContainerStarted","Data":"f63519959bd9e61df7c645b9ef003cd85e2e2df6fcca69f3cf60fabdc2f21095"}
Sep 10 23:40:26.234418 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:26.234359    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts" podStartSLOduration=1.234335558 podStartE2EDuration="1.234335558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:40:26.233840697 +0000 UTC m=+2346.925417881" watchObservedRunningTime="2026-09-10 23:40:26.234335558 +0000 UTC m=+2346.925912743"
Sep 10 23:40:31.952080 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:31.952036    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:31.952576 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:31.952098    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:31.964747 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:31.964719    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:32.259727 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:32.259639    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:40:35.576417 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:35.576375    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:35.576821 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:35.576432    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:35.589561 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:35.589536    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:40:36.282001 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:40:36.281968    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:41:06.212035 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:06.212002    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"]
Sep 10 23:41:07.581047 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:07.581010    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"]
Sep 10 23:41:20.000246 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:20.000217    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:41:20.003836 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:20.003814    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b88ks_df881283-5594-424d-98e4-e75fec3b7d19/ovn-acl-logging/0.log"
Sep 10 23:41:20.510932 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:20.510901    2563 scope.go:117] "RemoveContainer" containerID="c549c4a8d59c06d2e0f6af1eba7349cc732fe01b7d941e51ff579574fb4fed38"
Sep 10 23:41:21.231617 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.231557    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687" podUID="3e3aaa50-238e-4b71-be38-73b557d391d3" containerName="main" containerID="cri-o://07b464bc9ac3cb3a7b7b4b8a2d363ab6541fdd1b6eb97a6fbe29fe8552d2dd04" gracePeriod=45
Sep 10 23:41:21.477260 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.477235    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:41:21.495542 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.495441    2563 generic.go:356] "Generic (PLEG): container finished" podID="3e3aaa50-238e-4b71-be38-73b557d391d3" containerID="07b464bc9ac3cb3a7b7b4b8a2d363ab6541fdd1b6eb97a6fbe29fe8552d2dd04" exitCode=0
Sep 10 23:41:21.495542 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.495506    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687" event={"ID":"3e3aaa50-238e-4b71-be38-73b557d391d3","Type":"ContainerDied","Data":"07b464bc9ac3cb3a7b7b4b8a2d363ab6541fdd1b6eb97a6fbe29fe8552d2dd04"}
Sep 10 23:41:21.495542 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.495517    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"
Sep 10 23:41:21.495824 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.495545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687" event={"ID":"3e3aaa50-238e-4b71-be38-73b557d391d3","Type":"ContainerDied","Data":"d5db8ff5099331e703b3f320f0517b3203e13554a6dd99b1897c59bb392083c7"}
Sep 10 23:41:21.495824 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.495566    2563 scope.go:117] "RemoveContainer" containerID="07b464bc9ac3cb3a7b7b4b8a2d363ab6541fdd1b6eb97a6fbe29fe8552d2dd04"
Sep 10 23:41:21.504576 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.504555    2563 scope.go:117] "RemoveContainer" containerID="07b464bc9ac3cb3a7b7b4b8a2d363ab6541fdd1b6eb97a6fbe29fe8552d2dd04"
Sep 10 23:41:21.504916 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:41:21.504890    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b464bc9ac3cb3a7b7b4b8a2d363ab6541fdd1b6eb97a6fbe29fe8552d2dd04\": container with ID starting with 07b464bc9ac3cb3a7b7b4b8a2d363ab6541fdd1b6eb97a6fbe29fe8552d2dd04 not found: ID does not exist" containerID="07b464bc9ac3cb3a7b7b4b8a2d363ab6541fdd1b6eb97a6fbe29fe8552d2dd04"
Sep 10 23:41:21.504978 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.504929    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b464bc9ac3cb3a7b7b4b8a2d363ab6541fdd1b6eb97a6fbe29fe8552d2dd04"} err="failed to get container status \"07b464bc9ac3cb3a7b7b4b8a2d363ab6541fdd1b6eb97a6fbe29fe8552d2dd04\": rpc error: code = NotFound desc = could not find container \"07b464bc9ac3cb3a7b7b4b8a2d363ab6541fdd1b6eb97a6fbe29fe8552d2dd04\": container with ID starting with 07b464bc9ac3cb3a7b7b4b8a2d363ab6541fdd1b6eb97a6fbe29fe8552d2dd04 not found: ID does not exist"
Sep 10 23:41:21.531347 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.531316    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-home\") pod \"3e3aaa50-238e-4b71-be38-73b557d391d3\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") "
Sep 10 23:41:21.531485 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.531362    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-model-cache\") pod \"3e3aaa50-238e-4b71-be38-73b557d391d3\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") "
Sep 10 23:41:21.531485 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.531422    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-tmp-dir\") pod \"3e3aaa50-238e-4b71-be38-73b557d391d3\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") "
Sep 10 23:41:21.531485 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.531479    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vc6h\" (UniqueName: \"kubernetes.io/projected/3e3aaa50-238e-4b71-be38-73b557d391d3-kube-api-access-8vc6h\") pod \"3e3aaa50-238e-4b71-be38-73b557d391d3\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") "
Sep 10 23:41:21.531697 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.531564    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-dshm\") pod \"3e3aaa50-238e-4b71-be38-73b557d391d3\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") "
Sep 10 23:41:21.531697 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.531598    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3aaa50-238e-4b71-be38-73b557d391d3-tls-certs\") pod \"3e3aaa50-238e-4b71-be38-73b557d391d3\" (UID: \"3e3aaa50-238e-4b71-be38-73b557d391d3\") "
Sep 10 23:41:21.531697 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.531616    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-home" (OuterVolumeSpecName: "home") pod "3e3aaa50-238e-4b71-be38-73b557d391d3" (UID: "3e3aaa50-238e-4b71-be38-73b557d391d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:41:21.531884 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.531658    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e3aaa50-238e-4b71-be38-73b557d391d3" (UID: "3e3aaa50-238e-4b71-be38-73b557d391d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:41:21.531884 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.531729    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e3aaa50-238e-4b71-be38-73b557d391d3" (UID: "3e3aaa50-238e-4b71-be38-73b557d391d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:41:21.531994 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.531916    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:41:21.531994 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.531948    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:41:21.531994 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.531958    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:41:21.533820 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.533792    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e3aaa50-238e-4b71-be38-73b557d391d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e3aaa50-238e-4b71-be38-73b557d391d3" (UID: "3e3aaa50-238e-4b71-be38-73b557d391d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:41:21.533820 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.533803    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3aaa50-238e-4b71-be38-73b557d391d3-kube-api-access-8vc6h" (OuterVolumeSpecName: "kube-api-access-8vc6h") pod "3e3aaa50-238e-4b71-be38-73b557d391d3" (UID: "3e3aaa50-238e-4b71-be38-73b557d391d3"). InnerVolumeSpecName "kube-api-access-8vc6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:41:21.533820 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.533796    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-dshm" (OuterVolumeSpecName: "dshm") pod "3e3aaa50-238e-4b71-be38-73b557d391d3" (UID: "3e3aaa50-238e-4b71-be38-73b557d391d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:41:21.632430 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.632387    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vc6h\" (UniqueName: \"kubernetes.io/projected/3e3aaa50-238e-4b71-be38-73b557d391d3-kube-api-access-8vc6h\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:41:21.632430 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.632423    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3aaa50-238e-4b71-be38-73b557d391d3-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:41:21.632430 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.632434    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3aaa50-238e-4b71-be38-73b557d391d3-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:41:21.817317 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.817281    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"]
Sep 10 23:41:21.818903 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:21.818882    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-7baf1b07-kserve-9cc64867-p2687"]
Sep 10 23:41:22.600945 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.600881    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts" podUID="f1a55d86-e0a3-475a-bfca-9683afdea519" containerName="main" containerID="cri-o://60c0681b9f00c02b1252eac9a42ad0a3ce4fba6406cd409a3d74376eb1e7d8a2" gracePeriod=45
Sep 10 23:41:22.847356 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.847330    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:41:22.942723 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.942635    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-dshm\") pod \"f1a55d86-e0a3-475a-bfca-9683afdea519\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") "
Sep 10 23:41:22.942723 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.942685    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-home\") pod \"f1a55d86-e0a3-475a-bfca-9683afdea519\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") "
Sep 10 23:41:22.942723 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.942727    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-tmp-dir\") pod \"f1a55d86-e0a3-475a-bfca-9683afdea519\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") "
Sep 10 23:41:22.943018 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.942759    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffbb9\" (UniqueName: \"kubernetes.io/projected/f1a55d86-e0a3-475a-bfca-9683afdea519-kube-api-access-ffbb9\") pod \"f1a55d86-e0a3-475a-bfca-9683afdea519\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") "
Sep 10 23:41:22.943018 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.942778    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-model-cache\") pod \"f1a55d86-e0a3-475a-bfca-9683afdea519\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") "
Sep 10 23:41:22.943018 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.942795    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1a55d86-e0a3-475a-bfca-9683afdea519-tls-certs\") pod \"f1a55d86-e0a3-475a-bfca-9683afdea519\" (UID: \"f1a55d86-e0a3-475a-bfca-9683afdea519\") "
Sep 10 23:41:22.943018 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.942948    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-home" (OuterVolumeSpecName: "home") pod "f1a55d86-e0a3-475a-bfca-9683afdea519" (UID: "f1a55d86-e0a3-475a-bfca-9683afdea519"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:41:22.943229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.943069    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1a55d86-e0a3-475a-bfca-9683afdea519" (UID: "f1a55d86-e0a3-475a-bfca-9683afdea519"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:41:22.943229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.943080    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1a55d86-e0a3-475a-bfca-9683afdea519" (UID: "f1a55d86-e0a3-475a-bfca-9683afdea519"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:41:22.943229 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.943124    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-home\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:41:22.945002 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.944973    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-dshm" (OuterVolumeSpecName: "dshm") pod "f1a55d86-e0a3-475a-bfca-9683afdea519" (UID: "f1a55d86-e0a3-475a-bfca-9683afdea519"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:41:22.945122 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.945030    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a55d86-e0a3-475a-bfca-9683afdea519-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1a55d86-e0a3-475a-bfca-9683afdea519" (UID: "f1a55d86-e0a3-475a-bfca-9683afdea519"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:41:22.945122 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:22.945057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a55d86-e0a3-475a-bfca-9683afdea519-kube-api-access-ffbb9" (OuterVolumeSpecName: "kube-api-access-ffbb9") pod "f1a55d86-e0a3-475a-bfca-9683afdea519" (UID: "f1a55d86-e0a3-475a-bfca-9683afdea519"). InnerVolumeSpecName "kube-api-access-ffbb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:41:23.043952 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.043909    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-dshm\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:41:23.043952 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.043945    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-tmp-dir\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:41:23.043952 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.043961    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffbb9\" (UniqueName: \"kubernetes.io/projected/f1a55d86-e0a3-475a-bfca-9683afdea519-kube-api-access-ffbb9\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:41:23.044201 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.043973    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a55d86-e0a3-475a-bfca-9683afdea519-model-cache\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:41:23.044201 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.043984    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1a55d86-e0a3-475a-bfca-9683afdea519-tls-certs\") on node \"ip-10-0-130-246.ec2.internal\" DevicePath \"\""
Sep 10 23:41:23.507347 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.507315    2563 generic.go:356] "Generic (PLEG): container finished" podID="f1a55d86-e0a3-475a-bfca-9683afdea519" containerID="60c0681b9f00c02b1252eac9a42ad0a3ce4fba6406cd409a3d74376eb1e7d8a2" exitCode=0
Sep 10 23:41:23.507543 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.507355    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"
Sep 10 23:41:23.507543 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.507408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts" event={"ID":"f1a55d86-e0a3-475a-bfca-9683afdea519","Type":"ContainerDied","Data":"60c0681b9f00c02b1252eac9a42ad0a3ce4fba6406cd409a3d74376eb1e7d8a2"}
Sep 10 23:41:23.507543 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.507449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts" event={"ID":"f1a55d86-e0a3-475a-bfca-9683afdea519","Type":"ContainerDied","Data":"f63519959bd9e61df7c645b9ef003cd85e2e2df6fcca69f3cf60fabdc2f21095"}
Sep 10 23:41:23.507543 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.507473    2563 scope.go:117] "RemoveContainer" containerID="60c0681b9f00c02b1252eac9a42ad0a3ce4fba6406cd409a3d74376eb1e7d8a2"
Sep 10 23:41:23.517070 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.517053    2563 scope.go:117] "RemoveContainer" containerID="60c0681b9f00c02b1252eac9a42ad0a3ce4fba6406cd409a3d74376eb1e7d8a2"
Sep 10 23:41:23.517331 ip-10-0-130-246 kubenswrapper[2563]: E0910 23:41:23.517313    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c0681b9f00c02b1252eac9a42ad0a3ce4fba6406cd409a3d74376eb1e7d8a2\": container with ID starting with 60c0681b9f00c02b1252eac9a42ad0a3ce4fba6406cd409a3d74376eb1e7d8a2 not found: ID does not exist" containerID="60c0681b9f00c02b1252eac9a42ad0a3ce4fba6406cd409a3d74376eb1e7d8a2"
Sep 10 23:41:23.517376 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.517341    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c0681b9f00c02b1252eac9a42ad0a3ce4fba6406cd409a3d74376eb1e7d8a2"} err="failed to get container status \"60c0681b9f00c02b1252eac9a42ad0a3ce4fba6406cd409a3d74376eb1e7d8a2\": rpc error: code = NotFound desc = could not find container \"60c0681b9f00c02b1252eac9a42ad0a3ce4fba6406cd409a3d74376eb1e7d8a2\": container with ID starting with 60c0681b9f00c02b1252eac9a42ad0a3ce4fba6406cd409a3d74376eb1e7d8a2 not found: ID does not exist"
Sep 10 23:41:23.528379 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.528352    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"]
Sep 10 23:41:23.532655 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.532633    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-7baf1b07-kserve-7644866dc5-7msts"]
Sep 10 23:41:23.814411 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.814368    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e3aaa50-238e-4b71-be38-73b557d391d3" path="/var/lib/kubelet/pods/3e3aaa50-238e-4b71-be38-73b557d391d3/volumes"
Sep 10 23:41:23.814985 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:23.814964    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1a55d86-e0a3-475a-bfca-9683afdea519" path="/var/lib/kubelet/pods/f1a55d86-e0a3-475a-bfca-9683afdea519/volumes"
Sep 10 23:41:52.169462 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:52.169428    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7555cfcc9b-7rpzq_be49a481-46c0-41f9-b5f3-ee9f4e4e91ad/authorino/0.log"
Sep 10 23:41:58.006598 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:58.006567    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5b5b6476-7qp5k_8a9fe207-ecdb-494e-8bce-4fbd147f8ef6/copy-manifests/0.log"
Sep 10 23:41:58.012517 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:58.012483    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5b5b6476-7qp5k_8a9fe207-ecdb-494e-8bce-4fbd147f8ef6/manager/0.log"
Sep 10 23:41:58.680777 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:58.680749    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818035-d7ld4_ec78084d-b577-44c4-9a7d-e9a3f2b19bde/cleanup/0.log"
Sep 10 23:41:58.787957 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:58.787927    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818050-h78xl_a643cba3-f7d7-4070-a8d9-df6cf2f54445/cleanup/0.log"
Sep 10 23:41:59.624724 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:59.624678    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7555cfcc9b-7rpzq_be49a481-46c0-41f9-b5f3-ee9f4e4e91ad/authorino/0.log"
Sep 10 23:41:59.954161 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:41:59.954086    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bd4s9_559d95fc-38d9-4e28-a02d-6d577abbefdd/kuadrant-console-plugin/0.log"
Sep 10 23:42:00.307066 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:00.306976    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-lqn4f_4004b71a-b778-433a-a33b-9c089d353e88/manager/0.log"
Sep 10 23:42:00.868874 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:00.868817    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d6c69d6b7-stgvr_a46819a0-33b2-4794-8112-139132fb0102/kube-auth-proxy/0.log"
Sep 10 23:42:01.318138 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:01.318100    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mcn47_a2dc4813-36ba-4324-bf13-4e83f11ded1d/payload-processing/0.log"
Sep 10 23:42:01.425010 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:01.424958    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8484fd4884-9th5k_810a3f8a-76ae-4456-82d1-2f50e10721ac/router/0.log"
Sep 10 23:42:01.763918 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:01.763817    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-qv9qc_d38e4409-bcc9-4c8f-97d5-072a43d512c8/main/0.log"
Sep 10 23:42:07.942084 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:07.942047    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5xqw8/must-gather-f44jd"]
Sep 10 23:42:07.942471 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:07.942411    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1a55d86-e0a3-475a-bfca-9683afdea519" containerName="main"
Sep 10 23:42:07.942471 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:07.942423    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1a55d86-e0a3-475a-bfca-9683afdea519" containerName="main"
Sep 10 23:42:07.942471 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:07.942444    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e3aaa50-238e-4b71-be38-73b557d391d3" containerName="main"
Sep 10 23:42:07.942471 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:07.942452    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e3aaa50-238e-4b71-be38-73b557d391d3" containerName="main"
Sep 10 23:42:07.942668 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:07.942523    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1a55d86-e0a3-475a-bfca-9683afdea519" containerName="main"
Sep 10 23:42:07.942668 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:07.942531    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e3aaa50-238e-4b71-be38-73b557d391d3" containerName="main"
Sep 10 23:42:07.945966 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:07.945945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5xqw8/must-gather-f44jd"
Sep 10 23:42:07.948693 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:07.948674    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5xqw8\"/\"default-dockercfg-8m8wk\""
Sep 10 23:42:07.949210 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:07.949193    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5xqw8\"/\"openshift-service-ca.crt\""
Sep 10 23:42:07.949368 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:07.949353    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5xqw8\"/\"kube-root-ca.crt\""
Sep 10 23:42:07.954742 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:07.954718    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5xqw8/must-gather-f44jd"]
Sep 10 23:42:08.070184 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:08.070143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fd8e995-d5de-416c-873d-35bfde26ca57-must-gather-output\") pod \"must-gather-f44jd\" (UID: \"0fd8e995-d5de-416c-873d-35bfde26ca57\") " pod="openshift-must-gather-5xqw8/must-gather-f44jd"
Sep 10 23:42:08.070184 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:08.070187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8lws\" (UniqueName: \"kubernetes.io/projected/0fd8e995-d5de-416c-873d-35bfde26ca57-kube-api-access-w8lws\") pod \"must-gather-f44jd\" (UID: \"0fd8e995-d5de-416c-873d-35bfde26ca57\") " pod="openshift-must-gather-5xqw8/must-gather-f44jd"
Sep 10 23:42:08.171048 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:08.171012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fd8e995-d5de-416c-873d-35bfde26ca57-must-gather-output\") pod \"must-gather-f44jd\" (UID: \"0fd8e995-d5de-416c-873d-35bfde26ca57\") " pod="openshift-must-gather-5xqw8/must-gather-f44jd"
Sep 10 23:42:08.171048 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:08.171054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8lws\" (UniqueName: \"kubernetes.io/projected/0fd8e995-d5de-416c-873d-35bfde26ca57-kube-api-access-w8lws\") pod \"must-gather-f44jd\" (UID: \"0fd8e995-d5de-416c-873d-35bfde26ca57\") " pod="openshift-must-gather-5xqw8/must-gather-f44jd"
Sep 10 23:42:08.171377 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:08.171356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0fd8e995-d5de-416c-873d-35bfde26ca57-must-gather-output\") pod \"must-gather-f44jd\" (UID: \"0fd8e995-d5de-416c-873d-35bfde26ca57\") " pod="openshift-must-gather-5xqw8/must-gather-f44jd"
Sep 10 23:42:08.178837 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:08.178801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8lws\" (UniqueName: \"kubernetes.io/projected/0fd8e995-d5de-416c-873d-35bfde26ca57-kube-api-access-w8lws\") pod \"must-gather-f44jd\" (UID: \"0fd8e995-d5de-416c-873d-35bfde26ca57\") " pod="openshift-must-gather-5xqw8/must-gather-f44jd"
Sep 10 23:42:08.255252 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:08.255155    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5xqw8/must-gather-f44jd"
Sep 10 23:42:08.526168 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:08.526112    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5xqw8/must-gather-f44jd"]
Sep 10 23:42:08.528467 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:42:08.528439    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fd8e995_d5de_416c_873d_35bfde26ca57.slice/crio-891b6693466c1ca6a7a0aa4b48a96a0c90e39586ada04f3ce443656461ded0c7 WatchSource:0}: Error finding container 891b6693466c1ca6a7a0aa4b48a96a0c90e39586ada04f3ce443656461ded0c7: Status 404 returned error can't find the container with id 891b6693466c1ca6a7a0aa4b48a96a0c90e39586ada04f3ce443656461ded0c7
Sep 10 23:42:08.530287 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:08.530269    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:42:08.723603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:08.723556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xqw8/must-gather-f44jd" event={"ID":"0fd8e995-d5de-416c-873d-35bfde26ca57","Type":"ContainerStarted","Data":"891b6693466c1ca6a7a0aa4b48a96a0c90e39586ada04f3ce443656461ded0c7"}
Sep 10 23:42:09.735532 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:09.735408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xqw8/must-gather-f44jd" event={"ID":"0fd8e995-d5de-416c-873d-35bfde26ca57","Type":"ContainerStarted","Data":"09cfe89af5c23d17ae2339c8db895b258a0f7942e9cb8600e8881daa8d6283ba"}
Sep 10 23:42:09.735532 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:09.735472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xqw8/must-gather-f44jd" event={"ID":"0fd8e995-d5de-416c-873d-35bfde26ca57","Type":"ContainerStarted","Data":"3c97965ffd9e636784bc234e13d98e03e5c32bf9bbb6478a9ca7b1e8616894a4"}
Sep 10 23:42:09.751754 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:09.751690    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5xqw8/must-gather-f44jd" podStartSLOduration=1.947028858 podStartE2EDuration="2.751671393s" totalImagesPullingTime="804.642535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:09.749707559 +0000 UTC m=+2450.441284745" watchObservedRunningTime="2026-09-10 23:42:09.751671393 +0000 UTC m=+2450.443248577"
Sep 10 23:42:10.896029 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:10.895990    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xdms4_ab0193de-c9ef-4b5c-a865-bf880e0761d0/global-pull-secret-syncer/0.log"
Sep 10 23:42:11.040944 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:11.040911    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pgsk2_246cb358-713a-4429-aac2-8853a3d0dbdd/konnectivity-agent/0.log"
Sep 10 23:42:11.082977 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:11.082922    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-246.ec2.internal_8749419d048b9a7948ba1036740398d2/haproxy/0.log"
Sep 10 23:42:16.065903 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:16.065428    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7555cfcc9b-7rpzq_be49a481-46c0-41f9-b5f3-ee9f4e4e91ad/authorino/0.log"
Sep 10 23:42:16.187086 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:16.187048    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-bd4s9_559d95fc-38d9-4e28-a02d-6d577abbefdd/kuadrant-console-plugin/0.log"
Sep 10 23:42:16.334020 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:16.333909    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-lqn4f_4004b71a-b778-433a-a33b-9c089d353e88/manager/0.log"
Sep 10 23:42:18.032562 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.032532    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8d54bd73-9bf1-41b3-b40e-cb3320ab429e/alertmanager/0.log"
Sep 10 23:42:18.057849 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.057814    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8d54bd73-9bf1-41b3-b40e-cb3320ab429e/config-reloader/0.log"
Sep 10 23:42:18.080235 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.080197    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8d54bd73-9bf1-41b3-b40e-cb3320ab429e/kube-rbac-proxy-web/0.log"
Sep 10 23:42:18.105829 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.105659    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8d54bd73-9bf1-41b3-b40e-cb3320ab429e/kube-rbac-proxy/0.log"
Sep 10 23:42:18.127603 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.127575    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8d54bd73-9bf1-41b3-b40e-cb3320ab429e/kube-rbac-proxy-metric/0.log"
Sep 10 23:42:18.149716 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.149639    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8d54bd73-9bf1-41b3-b40e-cb3320ab429e/prom-label-proxy/0.log"
Sep 10 23:42:18.172302 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.172275    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8d54bd73-9bf1-41b3-b40e-cb3320ab429e/init-config-reloader/0.log"
Sep 10 23:42:18.224069 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.224017    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-bwkkv_d7a69d1d-f105-49eb-ad0e-808c10d9adae/cluster-monitoring-operator/0.log"
Sep 10 23:42:18.277536 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.277475    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-9jcw2_2317e5b0-b1d1-480b-baa0-944b0c684a4f/kube-state-metrics/0.log"
Sep 10 23:42:18.297038 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.296965    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-9jcw2_2317e5b0-b1d1-480b-baa0-944b0c684a4f/kube-rbac-proxy-main/0.log"
Sep 10 23:42:18.317528 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.317501    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-9jcw2_2317e5b0-b1d1-480b-baa0-944b0c684a4f/kube-rbac-proxy-self/0.log"
Sep 10 23:42:18.374707 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.374674    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-qjn5w_d5ba300f-7259-4027-b26d-4eab1b9bea4d/monitoring-plugin/0.log"
Sep 10 23:42:18.415325 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.415297    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jhbpm_f3b57650-50d3-4c2e-9480-a4edf504efcd/node-exporter/0.log"
Sep 10 23:42:18.436644 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.436616    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jhbpm_f3b57650-50d3-4c2e-9480-a4edf504efcd/kube-rbac-proxy/0.log"
Sep 10 23:42:18.459221 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.459196    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jhbpm_f3b57650-50d3-4c2e-9480-a4edf504efcd/init-textfile/0.log"
Sep 10 23:42:18.662892 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.662849    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-wpx66_bc9eb081-6a81-4a89-a50f-b705af231db2/kube-rbac-proxy-main/0.log"
Sep 10 23:42:18.688718 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.688693    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-wpx66_bc9eb081-6a81-4a89-a50f-b705af231db2/kube-rbac-proxy-self/0.log"
Sep 10 23:42:18.711669 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.711638    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-wpx66_bc9eb081-6a81-4a89-a50f-b705af231db2/openshift-state-metrics/0.log"
Sep 10 23:42:18.761176 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.761138    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ccdee23-a231-44be-a1d4-55ff61a2afc4/prometheus/0.log"
Sep 10 23:42:18.781219 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.781181    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ccdee23-a231-44be-a1d4-55ff61a2afc4/config-reloader/0.log"
Sep 10 23:42:18.805029 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.804995    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ccdee23-a231-44be-a1d4-55ff61a2afc4/thanos-sidecar/0.log"
Sep 10 23:42:18.832392 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.832358    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ccdee23-a231-44be-a1d4-55ff61a2afc4/kube-rbac-proxy-web/0.log"
Sep 10 23:42:18.856695 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.856668    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ccdee23-a231-44be-a1d4-55ff61a2afc4/kube-rbac-proxy/0.log"
Sep 10 23:42:18.877919 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.877892    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ccdee23-a231-44be-a1d4-55ff61a2afc4/kube-rbac-proxy-thanos/0.log"
Sep 10 23:42:18.899896 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:18.899865    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5ccdee23-a231-44be-a1d4-55ff61a2afc4/init-config-reloader/0.log"
Sep 10 23:42:19.005761 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:19.005681    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-578d599455-5jl7t_7554854d-c19c-4597-9046-6fc8c32ec82f/telemeter-client/0.log"
Sep 10 23:42:19.030193 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:19.030155    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-578d599455-5jl7t_7554854d-c19c-4597-9046-6fc8c32ec82f/reload/0.log"
Sep 10 23:42:19.051445 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:19.051415    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-578d599455-5jl7t_7554854d-c19c-4597-9046-6fc8c32ec82f/kube-rbac-proxy/0.log"
Sep 10 23:42:20.531081 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:20.531037    2563 scope.go:117] "RemoveContainer" containerID="225f9528e27f5165409e5610f6a24efcf22aa78a82ee72def84a2e57511c4922"
Sep 10 23:42:20.548121 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:20.547951    2563 scope.go:117] "RemoveContainer" containerID="b08369da38944ae27951d2d90e2c3a4a65db174d5dc9453950c77e4d3aa2fdb1"
Sep 10 23:42:21.775889 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:21.775849    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"]
Sep 10 23:42:21.781869 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:21.781836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:21.790204 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:21.790066    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"]
Sep 10 23:42:21.927220 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:21.927180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-lib-modules\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:21.927523 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:21.927503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-proc\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:21.927705 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:21.927689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzr8m\" (UniqueName: \"kubernetes.io/projected/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-kube-api-access-qzr8m\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:21.927862 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:21.927841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-sys\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:21.927998 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:21.927892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-podres\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:22.028992 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.028896    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzr8m\" (UniqueName: \"kubernetes.io/projected/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-kube-api-access-qzr8m\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:22.029169 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.029146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-sys\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:22.029243 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.029207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-podres\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:22.029442 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.029306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-lib-modules\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:22.029442 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.029360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-sys\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:22.029442 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.029366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-proc\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:22.029442 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.029417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-proc\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:22.029442 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.029419    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-podres\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:22.029755 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.029557    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-lib-modules\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:22.036316 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.036282    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzr8m\" (UniqueName: \"kubernetes.io/projected/20d4a5ba-32d1-48e3-87b0-13e67aa1a959-kube-api-access-qzr8m\") pod \"perf-node-gather-daemonset-g4rng\" (UID: \"20d4a5ba-32d1-48e3-87b0-13e67aa1a959\") " pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:22.098536 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.098465    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:22.410301 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.407171    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"]
Sep 10 23:42:22.410788 ip-10-0-130-246 kubenswrapper[2563]: W0910 23:42:22.410759    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod20d4a5ba_32d1_48e3_87b0_13e67aa1a959.slice/crio-6696cfc657e52815b70413535ba21e6a65af65db1af90e67ba05d83c43b4cfd2 WatchSource:0}: Error finding container 6696cfc657e52815b70413535ba21e6a65af65db1af90e67ba05d83c43b4cfd2: Status 404 returned error can't find the container with id 6696cfc657e52815b70413535ba21e6a65af65db1af90e67ba05d83c43b4cfd2
Sep 10 23:42:22.829726 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.829685    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng" event={"ID":"20d4a5ba-32d1-48e3-87b0-13e67aa1a959","Type":"ContainerStarted","Data":"631cb8b1d80fe9bcdb7fce9d11343b8578b56626ad0a87e1e5029343e1017a92"}
Sep 10 23:42:22.830164 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.829746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng" event={"ID":"20d4a5ba-32d1-48e3-87b0-13e67aa1a959","Type":"ContainerStarted","Data":"6696cfc657e52815b70413535ba21e6a65af65db1af90e67ba05d83c43b4cfd2"}
Sep 10 23:42:22.830164 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.829845    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:22.845029 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:22.844972    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng" podStartSLOduration=1.844951489 podStartE2EDuration="1.844951489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:22.844061552 +0000 UTC m=+2463.535638748" watchObservedRunningTime="2026-09-10 23:42:22.844951489 +0000 UTC m=+2463.536528674"
Sep 10 23:42:23.724512 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:23.724469    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qq99b_d3aff66a-de8c-4842-902b-fe0015f1ae4b/dns/0.log"
Sep 10 23:42:23.744680 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:23.744650    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qq99b_d3aff66a-de8c-4842-902b-fe0015f1ae4b/kube-rbac-proxy/0.log"
Sep 10 23:42:23.765722 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:23.765689    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5j59t_02e28b7f-e193-411b-875c-e475adab6c34/dns-node-resolver/0.log"
Sep 10 23:42:24.396193 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:24.396161    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n6rbk_bc7dd59a-fbd1-49b2-b886-ee2429896ddc/node-ca/0.log"
Sep 10 23:42:25.603752 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:25.603724    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d6c69d6b7-stgvr_a46819a0-33b2-4794-8112-139132fb0102/kube-auth-proxy/0.log"
Sep 10 23:42:25.730037 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:25.730007    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mcn47_a2dc4813-36ba-4324-bf13-4e83f11ded1d/payload-processing/0.log"
Sep 10 23:42:25.752377 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:25.752340    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8484fd4884-9th5k_810a3f8a-76ae-4456-82d1-2f50e10721ac/router/0.log"
Sep 10 23:42:26.660752 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:26.660713    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hx99c_43c78e11-d941-4f95-99b2-abd00726363e/serve-healthcheck-canary/0.log"
Sep 10 23:42:27.166470 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:27.166440    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-96fls_77ac1f77-479a-4611-b90c-c5ffb2242a9f/insights-operator/0.log"
Sep 10 23:42:27.167448 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:27.167425    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-96fls_77ac1f77-479a-4611-b90c-c5ffb2242a9f/insights-operator/1.log"
Sep 10 23:42:27.249623 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:27.249598    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g8zrr_5b0e7eaf-7d3d-481d-87ae-1773ea4fc258/kube-rbac-proxy/0.log"
Sep 10 23:42:27.268750 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:27.268724    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g8zrr_5b0e7eaf-7d3d-481d-87ae-1773ea4fc258/exporter/0.log"
Sep 10 23:42:27.287575 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:27.287550    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g8zrr_5b0e7eaf-7d3d-481d-87ae-1773ea4fc258/extractor/0.log"
Sep 10 23:42:28.849542 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:28.849483    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5xqw8/perf-node-gather-daemonset-g4rng"
Sep 10 23:42:29.901046 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:29.901012    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5b5b6476-7qp5k_8a9fe207-ecdb-494e-8bce-4fbd147f8ef6/manager/0.log"
Sep 10 23:42:29.919016 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:29.918992    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f5b5b6476-7qp5k_8a9fe207-ecdb-494e-8bce-4fbd147f8ef6/copy-manifests/0.log"
Sep 10 23:42:31.368091 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:31.368041    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-559dcbdd76-4h95d_a8c3c281-3f33-4c3c-9ac6-99ce91f7556a/manager/0.log"
Sep 10 23:42:32.620390 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:32.620348    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d6c69d6b7-stgvr_a46819a0-33b2-4794-8112-139132fb0102/kube-auth-proxy/0.log"
Sep 10 23:42:32.722816 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:32.722780    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mcn47_a2dc4813-36ba-4324-bf13-4e83f11ded1d/payload-processing/0.log"
Sep 10 23:42:32.737006 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:32.736980    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8484fd4884-9th5k_810a3f8a-76ae-4456-82d1-2f50e10721ac/router/0.log"
Sep 10 23:42:34.629244 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:34.629206    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d6c69d6b7-stgvr_a46819a0-33b2-4794-8112-139132fb0102/kube-auth-proxy/0.log"
Sep 10 23:42:34.722670 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:34.722640    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mcn47_a2dc4813-36ba-4324-bf13-4e83f11ded1d/payload-processing/0.log"
Sep 10 23:42:34.741011 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:34.740979    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8484fd4884-9th5k_810a3f8a-76ae-4456-82d1-2f50e10721ac/router/0.log"
Sep 10 23:42:38.091368 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:38.091334    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-md4pm_0b14fb8d-3467-4ca7-9200-d6ae4692aa22/kube-multus-additional-cni-plugins/0.log"
Sep 10 23:42:38.110816 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:38.110788    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-md4pm_0b14fb8d-3467-4ca7-9200-d6ae4692aa22/egress-router-binary-copy/0.log"
Sep 10 23:42:38.129145 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:38.129116    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-md4pm_0b14fb8d-3467-4ca7-9200-d6ae4692aa22/cni-plugins/0.log"
Sep 10 23:42:38.148126 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:38.148101    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-md4pm_0b14fb8d-3467-4ca7-9200-d6ae4692aa22/bond-cni-plugin/0.log"
Sep 10 23:42:38.166900 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:38.166878    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-md4pm_0b14fb8d-3467-4ca7-9200-d6ae4692aa22/routeoverride-cni/0.log"
Sep 10 23:42:38.185969 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:38.185947    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-md4pm_0b14fb8d-3467-4ca7-9200-d6ae4692aa22/whereabouts-cni-bincopy/0.log"
Sep 10 23:42:38.204558 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:38.204533    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-md4pm_0b14fb8d-3467-4ca7-9200-d6ae4692aa22/whereabouts-cni/0.log"
Sep 10 23:42:38.404893 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:38.404866    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ctqkx_21f78481-f131-45b0-ba1d-498e4f3d76bd/kube-multus/0.log"
Sep 10 23:42:38.547118 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:38.547086    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v8sx8_54f0cb94-e7d5-4b86-8602-dd8b46138dab/network-metrics-daemon/0.log"
Sep 10 23:42:38.561197 ip-10-0-130-246 kubenswrapper[2563]: I0910 23:42:38.561170    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v8sx8_54f0cb94-e7d5-4b86-8602-dd8b46138dab/kube-rbac-proxy/0.log"